JavisDiT: Joint Audio-Video Diffusion Transformer with Hierarchical Spatio-Temporal Prior Synchronization
Authors: Kai Liu
Wei Li
Lai Chen
Shengqiong Wu
Yanhao Zheng
Jiayi Ji
Fan Zhou
Jiebo Luo
Ziwei Liu
Hao Fei
Tat-Seng Chua
Venue: ICLR 2026 Poster (2026)
Keywords: Diffusion Transformer
Joint Audio-Video Generation
Text-to-Audio-Video Generation
Video Generation
Primary area: generative models
TL;DR: We introduce JavisDiT, a novel Joint Audio-Video Diffusion Transformer designed for synchronized audio-video generation (JAVG) from open-ended user prompts.

Abstract: This paper introduces JavisDiT, a novel Joint Audio-Video Diffusion Trans- former designed for synchronized audio-video generation (JAVG). Based on the powerful Diffusion Transformer (DiT) architecture, JavisDiT simultaneously generates high-quality audio and video content from open-ended user prompts in a unified framework. To ensure audio-video synchronization, we introduce a fine-grained spatio-temporal alignment mechanism through a Hierarchical Spatial-Temporal Synchronized Prior (HiST-Sypo) Estimator. This module extracts both global and fine-grained spatio-temporal priors, guiding the synchronization between the visual and auditory components. Furthermore, we propose a new benchmark, JavisBench, which consists of 10,140 high-quality text-captioned sounding videos and focuses on synchronization evaluation in diverse and complex real-world scenarios. Further, we specifically devise a robust metric for measuring the synchrony between generated audio-video pairs in real-world content. Experimental results demonstrate that JavisDiT significantly outperforms existing methods by ensuring both high-quality generation and precise synchronization, setting a new standard for JAVG tasks. Our code, model, and data are available at https://javisverse.github.io/JavisDiT-page/.

Published as a conference paper at ICLR 2026
JAVISDIT: JOINT AUDIO-VIDEO DIFFUSION TRANSFORMER WITH HIERARCHICAL SPATIO- TEMPORAL PRIOR SYNCHRONIZATION
Kai Liu 1,2 L Wei Li Lai Chen', Shengqiong Wu², Yanhao Zheng', Jiayi Ji²,
Fan Zhou', Jiebo Luo", Ziwei Liu', Hao Fei"; Tat-Seng Chua?
ZZeejang University, 2 National University of Singapore.
3 University of Science and Technology of China, Unniversity of Rochester, "Nanyang Technological University
ABSTRACT
This paper introduces JavisDiT, novel Joint Audio-Video Diffusion Trans- former designed for synchronized audio-vile viideo generation (JAVG). Based on the powerful Diffusion Transformer (DiT) architecture, JavisDiT simultaneously gener- ates high-quality audio and video content from open-ended user prompts in a unified framework. To ensure audio-video synchronization, we introduce fine- fine-grained spatio-temporal alignment mechanism through a Hierarchical Synchronized Prior (HiST-Sypo) Estimator. This module extracts both global and fine tme-grained spatio-temporal priors, guiding the synchronization between the visual and auditory components. Furthermore, we propose new benchmark, JavisBench, which consists of 140140 high-quality text-captioned sounding videos and focuses on synchronization evaluation diverse and complex real- world scenarios. Further, specifically devise robust metric for measuring the synchrony between generated audio-video pairs in real-world content. Ex- perimental results demonstrate that JavisDiT significantly outperforms existing methods by ensuring both high-quality generation and precise synchronization, setting new standard for JAVG tasks. Our code, model, and data are available at hhttts::/ http:://aaaieeereee.ttt.b....... wtw.bbb..oJJaviiiiiiiii page/
INTRODUCTION
In the field of Al-generated content (AIGC), multimodal generation that covers images, videos.
and audio has gained increasing attention (Lyu et al. 20224, Liu et al. 2024b Yang et al. 2024b
Lyu et al. 2025). where feafusioonbbsedd models (Ho et al., 2020 Liu et al. 2023b; Peebles &
Xie, 2023) have demonstrated remarkable performance. While early studies mainly focused on
single-modality generation, recent work has shifted toward the simultaneous generation of multiple modalities (Gadre et al. 2024 Wu et al.). Notably, synchronized audio and video generation (Ruan) cet al., 2023 Xing et al., 2024 Sun et al. 2024) has emerged as a crucial area of study. Audio and video are inherently interconnected in most world scenarios, making their joint generation highly valuable for applications, such as movie production and short video creation. Current approaches to synchronized audio-video generation can be broadly categorized into two types. The first involves asynchronous pipelines, where audio is generated first and then used to synthesize video Yarri et al.
2024 Jeong et al. 2023 ZZhang et al. 2025). vice versa (Comunità al., 2024 Zhang al.
2024 Xie et al.] 2024). The second type involves end-to-end Joint Audio Video Generation (namely
JAVG) (Ruan et al.] 2023 Xing et al. 2024 [Snn al. 2024), which, by avoiding noise accumulation
during cascading processes of the first type, has thus attracted more research attention. Overall, it is strongly and widely believed that promoting JAVG requires two equally critical criteria: (1) ensuring
high-quality generation of audio and video, and (2) pursuing precise synchronization between two modalities.
Equal contribution. Work done during Kai Liu's visiting period at NUS. Email: kail @ @zw.@eju.uuccn Corresponning author. Email: haofei7419c @ gmail.com

---

Published as a conference paper at ICLR 2026
On one hand, for high-quality audio-vides generation, the backbone modules of both branches need to be as strong as possible. Recent JAVG research (Ruan et al. 2023: Xing et al.) 2024 Sun et al.
2024)
attempts to take Diffusion Transformer (DiT) (Peebles & Xie 2023 as backbone architectures, due to
the remarkably enhanced performance for vision and even audio tasks. In particular, AV-DiT Wang
et al. 2024bb and MM-LDM (Sun al. (024) inherit an image DiT (Peebles & Xiel 2023 Esser
et al., 2024) to generate video and audio, where the fine-grained spatio-temporal modeling capability can be however limited. Concurrent works like Uniform (Zhao et al., 2025) and SyncFlow (Liu et al.)
2024a) take the enhanced STDiT3 (Zheng et al. 2024) blocks, but they either simply concatenate video/audio tokens or utilize mono-direecional video-to-audio adapter, lacking sufficient mutual information exchange between the two channels.
On the other hand, in terms of syn- chronization between the audio and video channels JAVG, extensive in- vestigations have been made to ex- plore more effective alignment strate- gies. However, most of these works still fall prey to insufficient model- ing of synchronization. For exam- ple, some emphasize coarse-grained temporal alignment by implement- ing straightforward parameter shar- ing (Ruan et al., 2023; Wang al.) 2024b] Zhao et al. (0255) or tempo- ral modulating (Wang et al.] 2024b] Liu et al., (222a) between the audio and video channels. Others focus on coarse-grained semantic alignment, such as
Input (Text prompt)
surny backyard, a robot tussles with mischievous black dog. their scutlle accompanied by mechonical whirs and playfal squeaks. Two aliens then appear behind the fence and begin talking.
Output (ppntal comporatly sweckronized soundling
Dog Playful squcaks
barking
Playful squends
Playful squwekks
Rabot Mechunical whirs
Machanical whire
Talking 4o0
(Xing et al.) 2024), which carries out simple audio- video embedding alignment, and MM- LDM (Sun al.) 2024) performing straightforward representation align- ment in the latent space of the diffusion model. These approaches fail to consider a fine-grained spatio-tempora alignment, which is essential for realistic synchronized audio-video generation. Specifically, in a realistic-soundin sounding video, all visual and auditory contents should be dictated by spatio-temporal characteristics of various objects, i.e., a) spatially distinct visual content (e.g., where events occur and how objects move), and b) temporally corresponding auditory attributes, such as source category and synchronized duration. We illustrate this in Fig. [ A dog and robot are playing on the ground, while an alien appears and starts to talk. In this scene, the robot's mechanical whirs and the dog's ' squeaks and barks keep consistent, and the alien's talking emerges later and continues to the end. Overall, the visual content and audio stay synchronized both temporally and spatially.
Figure 1: Given the text prompt, JAVG system generates a tempool-empporally synchronized sounding video. The sounds align precisely with the temporal progression of the actions.
To address the above dual challenges, this paper proposes novel niche- teche-taggetig JAVG system, called JavisDiT (cf. Fig. 2). First, Jav adopts the DiT architecture as the backbone, where the audio and video channels share AV-DIT blocks enable high-quality audio-video generation. Within JavisDiT, we design three infrastructural blocks: Spatio-Temponn Self-Attention, Coarse-Grained Cross-Attention, and Fine-Grained the fine-grained spatio-temporal alignment idea, we design a Hierarchical nized Prior (HiST-Sypo) estimation module. The HiST-Sypo Estimator hierarchically extracts the priors, such
Cross-Attention (ST-CorssAttn). To implement Synchro-
following from the input conditional prompt: global coarse- coarse-grained as the semantic framework of the overall sounding video; fine- fiee-graieed apatio-tempon priors, such as the distinct visual content triggered by various sounds and their corresponding temporal priors.
These global and fine-grained priors serve as spatio-temporal features, injected into different AV-DIT blocks to guide the spatial semantic and temporal synchronization of both audio and video. Built on JivisDiT, we further design a contrastive learning-based (Chen et al. 2020b) HiST-Sypo estimation strategy to learn robust spatial-temporal synchronized prior knowledge from large-scale data of sounding videos.
2

---

Published as a conference paper at ICLR 2026
Further, we observe that the existing JAVG evaluation benchmarks, e.g., Landscape (Lee al. 2022
and AIST+ (Li et al. 202I) suffer from certain limitations, including overly simplistic content and limited scene diversity (Mao et al. 2024). This creates evident gap compared to the complex audio-video content encountered in real-world applications, ie,, models (Ruan et al. 2023 Sun et al., 2024) trained there largely suffer from out-of-domain issue, thereby partially hindering the development and practical applicability on open environment. To bridge these gaps, we introduce new JAVG benchmark, called Javi BBenh.. After rigorous manual inspection, we collect total of 10,140 high-quality text-captioned sounding videos, which span over 5 dimensions with 19 scene categories. Notably, more than 50% of these videos feature highly complex and challenging scenarios, making the dataset more representative of diverse real-world applications. Meanwhile, we find that existing JAVG evaluation method can also limited by failing to adequately assess the JAVG systems on complex sounding videos. Thus, we devise a novel metric (JavisScore) based on temporal-awwe -warr semantic alignment mechanism.
Extensive experiments on both existing JAVG benchmarks and our JavisBench dataset demonstrate that JavisDiT significantly outperforms state-of-the-art methods in generating high-quality sounding videos. Our system is especially effective in handling complex-scene videos, thanks to the HiST-Sypo estimation mechanism. summary, our main contributions are as follows:
We introduce Javi novel JAVG model with hierarchical spatio-temporal prior estima- tion mechanism to achieve audio-video synchronization in both spatial and temporal dimensions. We contribute Javi sBench, a new large-scale JAVG benchmark dataset with challenging scenarios, along with robust metries to evaluate audio-video synchronization, offering a compre- hensive baseline for future research.
Empirically, our system achieves state-of thetthrrrtt performance across both existing closed-set and our open-world benchmarks, setting a new standard for JAVG.
PRELIMINARIES
Task Definition and Formulation. Joint Audio Video Generation (JAVG) requires diffusion models Go to simultancously generate videos v and corresponding audios a given a text input s. The forward process at timestamp t, either with DDPM (Ho al. 2020) or FlowMatching (Lipman et al., 2023 is formulated as:
1, a1-1) = Go (Ve, a,, 8,
(1)
V E R denotes video with T, frames and 3 RGB channels, where H, W are the frame height and width. On the other hand, a E TT Mx1 encodes audio in the image-like mel-spectrooram with T, temporal frames and I channel, where M is the frequency bins.
Alignment. For an ideal model G. the generated video and audio should maintain coarse-gre coarse-grained grained semantic alignment with the input text while further achieving Ene-grained spatio- temporal alignment between video v and audio a, which are formally defined as:
Spatial Alignment means the occurrence of specific event in a region of the video frame (at the H x W dimension in v), which corresponds to the appearance of matching frequency components in the audio (at the M dimension a).
Temporal Alignment means the onset or termination of an event at specific frame or timestamp the video (at the T, dimension in v), which must coincide with the corresponding start or stop of the response in the audio (at the T, dimension in a).
We focus on the joint generation of high-quality and spatio-temporally aligned video-audio pairs.
THE PROPOSED JAVISDIT SYSTEM
3.1 JAVISDIT MODEL ARCHITECTURE
Fig. 2 iillustrates the overall architecture. Spatio-temporal attention is employed for effective cross- modal alignment while ensuring high-quality generation for videos and audios. Each branch uses ST-SelfAm for modal aggregation, incorporates coarse text semanties via CrossAmn, integrates fine-grained spatio-temporal priors through ST-CrosSAth, and enhances video-audio fusion with Bi- Bi-CrossAtm. Here we introduce the core components displayed in Fig. (b):
Self-Attention. Given that both videos and audios possess spatial and temporal attributes (Ruan al. 2023), we employ cascaded spatio-temporal self-attention mechanism for
3

---

Published as a conference paper at ICLR 2026
In brokgaand with white picket fencs and small
ST-SelfAttn
Fine-Grained ST ST-CrossAttn
55.CC ViA Latent
[8,C+ VA Latent
cobetiic vehicle cumera-like
Text
Spatio-lzugieal
Audio Enceder
[(R [(++T).
Video
Spotkl
Pilor
Juato Laurs
MHSA
MICA
Sprisa Preur
Spatie-Tempera Temporal
Cowna-Crrarred Semantae-Prr
Tempo-Terporaal
[(R (8..)
[ -5.
Self-Aneeh Self-Amention
Self-Amention
MICA
Tompural Pricr
Grained
Grasee Cross Attention
[T. 5).0 55.c
[8.07 5).6)
Creea Cine--Atention
MM-BiCrostAmm
Ginened
Video Lateni
Audio Lateni
ST-CrossAtt
ST-CrossAmm
K.P
MM BiCrossAAm
Audio-to-video Cross-Alleniiee
FeedForward
FowdForward
XN
Video
(a) DiT bused Joint Sounding Video Generation Architecture
(b) Detailed Module Structure
Figure 2: On the left, present the overall DiT-based sounding video generation architecture of JavisDiT, consisting of a video generation branch, audio generation branch, HiST-Sypo Estimator module, and the MM-BiCrossAttn module. On the right, we illustrate the detailed structural design of Spatio-tempone Self-attention (ST-SelfAttn). Fine-grained Spatio-temporal Cross-attention (Fine- Grained ST-Cr,ssAttm), and Multi-Modality Bidirectional Cross-attention (MM-BiCrossAttn))
intra-modal information aggregation. As illustrated Fig. ()), MHSA is applied sequentially along the spatial ((I W) for v, .M for a) and temporal (T' for v. T. for a) dimensions. This efficiently achieves fine- graingr spatio-temporal modeling with reduced computational cost.
Spatio-Temporal Cross-Attention. For text prompt after injecting coarse semantics from the T5 encoder (Raffel et al. 2020) via vanilla cross attention, we estimate spatial and temporal priors with N₂ and N₂ learnable c-dimensional tokens by our ST-Prior Estimator (see details in Sec. As Fig. b)) shows, spatial and temporal priors guide cross-attention in both branches along the spatial and temporal dimensions, enabling unified, fine- fine-grained conditioning for video-audio synchronization.
Cross-Modallc Aedirectional-AAteention After aligning video and audio with ST-Prior, we incorpo- rate a bidirectional attention module (Liu et al. 2025) to enable direct cross-modal interactions. As Fig. ())) illustrates, after computing the attention matrix A between Go and kg. first multiply A Similarly, multiplying A (the transpose of A)
with D. to obtain the andio-to-vides cross-attentin with Ue yields the video-to-audio cross-attention. This mechanism enhances cross-modal information aggregation to facilitate high-quality joint audio-video generation.
3.2 HIERARCHICAL SPATIAL-TEMPON CATPPORRA..SNCCCRROCCOOO SYNCHRONIZED PRIOR ESTIMATOR
Unlike previous works, our JavisDiT derives two hierarchical conditions (priors) from texts: a global semantic prior for coarse event suggestion (what) and a fine-grained spatio-temporal prior (ST-Prior) specify event timing and location (when and where). These complementary priors unified into a HiST-Sypo Estimator, enabling precise synchronization between generated video and audio.
Coarse-Grained Spatio-Temporal Prior. Since the default semantic embeddings from T5 en- coder (Raffll et al. 2020) are strong enough to coarsely describe the overall sounding event, we simply reuse embeddings as our coarse- graiined spatio-temporal prior (or, semantic prior).
Text inputs typically provide coarse goaeeegrainddddeeee descriptions of events, egg., "a car starts its engine and leaves the screen"'. To enable fine- aeee-grannd conditioning, we estimate spatio-temporal priors: spatial prior specifies where events occur (e.g., "the car is in the top-left corner of screen""), ) and temporal prior defines when they start and end (e.g., "sound starts at 2s, exits at 7s, fades at 9s"). Instead of generating explicit prompts from LLMs (Liu et al. 2024b), we efficiently estimate spatio-temporal priors as latent token conditions to guide the diffusion process.
GinenGrained Pentio-Temporal Prior.
4

---

Published as a conference paper at ICLR 2026
Fine-Grained ST-Prior Estimation. As shown
Text Prompt
Video
Audio
in Fig. 3 for a given text input 8, we utilize
backeyard white picket fence and green grass.
the 77 hidden states from ImageBind's (Gird- har et al. 2023) text encoder, and use N₂- 32 spatial tokens P, and N₂ 32 temporal tokens Pr to query a 4-layer transformer P to extract spatio-temporal information. Since input text prompts usually may not specify an event's hap- pening time and location, it may almost occur and cease in arbitrary location and timing, and the same text 8 should correspondingly yield different ST-Priors (P&: Pt). To capture this variability, our ST-Prior Estimator P outputs the mean and variance of Gaussian distribu- tion, from which a plausible (Ps. P:) is sam- pled: (p&, Pi) c)) where is Gaus- sian noise. Furthermore, we carefully devised contrastive learning approach learn robust ST-Prior Estimator, which involves series of negative sample (asynchronous video-audio pairs) construction strategies and specifically loss functions. Implementation details are provided in the Sec. C.2
Imagehind Encoder
Audio Encoder
Encoder
F.P
Transformer Encoder
Spanial Query
Tomporai Quary
VA-Fuser
Transformer Decoder
Spartal
Temporal Prior
Vd-ipatial
Vd-Tenpomi
Contractrve Learning
Contrastive Learning
Figure 3: The framework of Spatio-temporal Prior Estimator with 4-layer transformer encoder-decoder (referring to the purple region). Contrastive learning is utilized for optimization.
3.3 MULTI-STAGE TRAINING STRATEGY
Our JavisDiT focuses on two core objectives: ensuring fine-grained spatio-temporal alignment between generated videos and audios. To do this, we leverage the pretrained weights of OpenSora (Zheng et al. (2024) for the video branch, and adopt three-stage training strategy for robust joint video-audio generation:
achieving high-quality single-modal generation and
1) Audio Pretraining. We initialize the audio branch with OpenSora's 's video branch weights and train it on 0.8M audio-text samples to ensure superior single-modal generation quality.
2) ST-Prior Training. We train the ST-Prior Estimator Ps using 0.6M synchronous text-video- audio triplets and synthesized asynchronous negative samples to strengthen representation.
3) JAVG Training. We freeze the video and audio branches' self-attention blocks and ST-Prior Estimator, training only the ST-CrossAttn and Bi-CrossAttn modules with 0.6M samples to enable synchronized video-audio generation.
addition, dynamic temporal masking (heng et al., 2024) enables flexible adaptation of JavisDiT to x-conditional tasks (v2a/a2v generation, image animation, video-audio extension, etc.). Details refer to Sec.
A CHALLENGING JAVISBENCE BENCHMARK
strong generative model must ensure diverse video content, audio types, and fine-grained spatio- temporal synchrony. However, current JAVG evaluation benchmarks fall short in encompassing the diversity of scenes, the complexity of real-world environments, and the presence of multiple sounding events within single audio-video instance, thereby constraining comprehensive and realistic evaluation (detailed in Sec. D.I). To combat this, we propose a more challenging benchmark featuring complex multi-event video-audio pairs (Sec. 4.1.. along with a robust synchronization- oriented metric tailored for evaluating fine- mee-grained spatio-temppral alignment (Sec. 4.2)
4.1 DATA CONSTRUCTION
Taxonomy, We design five evaluation dimensions from coarse to fine:
Event Scenario depicts scenarios where audio-visual events happen, such nature or industry.
Video Style describes the visual style of given videos, such as camera shooting or 2D animations.
Sound Type: the sounding type of given audios, such as sound effects music.
Spatial Composition defines the sounding subjects appearing in videos and audios, divided by
single multiple sounding subjects that exist.
Temporal Composition describes events' onsets and terminations in v-a pairs, distinguished by single multiple sounding sources that sequentially concurrently occur.
5

---

Published as a conference paper at ICLR 2026
Benchmark
#Sample
Scenarios
Categories
Event Scenario
Visual Style
Sound Type
Vintal
Inbeatil
Arbhan
Speech
A
20
(dancing)
1 (music)
Living
Biologicel
Nari
Camaua
Landscape
100
1 (natural)
1 (ambient)
Urfon
Medhaniel
Maskol
Spatial Composition
Temporal Composition
TAVGBench
3,000
Unknown Indiscriminate
Sugle
Singl
Offireenn
Soquerial
Ours
10,140
Diversified Hierarchical
Miligle
Figure 4: Left: Comparison between JavisBench and other benchmarks. TAVGBench (Mao et al. 2024)'s test set not yet released (Unknown). Right: Detailed category distribution of JavisBench.
Using GPT-4 (Achiam et al. 2023), we develop a hierarchical categorization system with 5 dimensions and 19 categories (Fig. 4). and the detailed definitions are presented in Sec. D.2
Data Curation. There are two data sources: (1) test sets of existing datasets (e.g., Landscape/AIST++ and FAVDBench (Shen et al. () and (2) YouTube videos uploaded between June and December 2024 to prevent data leakage (Ma et al. 2024). To collect YouTube videos, prompt GPT-4 to generate category-speciff keywords using our defined taxonomy, enabling efficient and targeted video collection while avoiding noisy data. After striet manual legal and ethical verification (see Sec. [.I]). Sec.[A.I) the above process yielded around 30K sounding video candidates. With several filtering tools ensure quality and diversity, we the advanced Qwen-family models (Yang et al.] 2024a Wang et 2024c Chu et al. 2024) to generate captions and categorize video-audio pairs into desired laxonomy, and finally curate 10.140 diverse and high-quality vidco-audio pairs for JavisBench- More details about data construction are presented in Sec. [33] We also randomly select 1,000 samples to form the version of JavisBench-mini for efficient evaluation.
Benchmark Statistics. Fig. 4 highlights JavisBench's ' contributions: (1) offering more diverse data compared to AIST++ (Li et al. 2021) and Landscape (Lee al. 2022), (2) providing a detailed taxonomy for comprehensive evaluations, surpassing TAVGBench (Mao et al. 2024). and (3) fea- turing the first evaluation benchmark focused on multi-event synchronization. As Fig. 4 shows, JavisBench covers diverse event scenarios, visual styles, and audio types, ensuring balanced distri- bution for greater diversity. Most realistic but under-represented scenarios like 2D/3D animations (25%) and industrial events (13%) are included. Furthermore, JavisBench emphasizes spatial and temporal complexity. 75% samples feature multiple sounding events, 28% involve sequential events, and 57% include simultaneous events. These diverse multi-event scenarios pose significant challenges for JAVG models, particularly in achieving accurate spatial and temporal alignment.
4.2 JAVISSCORE: MORE ROBUST JAVG METRIC
Motivation. AV-Align Yarri et al. 2024) is a widely-adopted JAVG metric, which uses video optical-flow estimation match the onset detected in audios measure temporal synchronization. However, AV-Align may struggle with complex scenarios (i.e., with multiple sounding events or subtle visual movements) and produce misleading results (see Sec. D.4p. Therefore, we propose more robust evaluation metric, namely JavisScore, to measure spatiotemporal synchronization in diverse, real-world contexts.
Implementation and Verification. Technically, we chunk each video-audio pair into several clips with 2-second window size and 1.5-second overlap, use ImageBind (Girdhar al. (023) compute the audio-visual synchronization for each segment, and average the scores as the final metric:
W
Slavis
o (V.)
(2)
W
1=1
where E, and Eg are vision and audio encoders, A, and Vi2 are audio and video segments in i-th window, W is the number of windows, and is specifically designed synchronization measure for audeo-audio segments (see Sec. D.4p. Motivated by Mao et al. (2024). we calculate similarities between all frames and the audio within each segment, and select the 40% least synchronized frames to obtain the score of the current window. Sec. D.4 also presents a annotated evaluation dataset with 3,000 samples to verify JavisScore's 's efficacy against previous metrics.

---

Published as a conference paper at ICLR 2026
Table 1: Main results on proposed JavisBench. Models are evaluated with 4-second videos at 240P/24fps and audio at 16kHz. JivisDiT comprehensively outperforms or gets par with currently available SoTA approaches. Best and secondary results are marked bold and underline respectively.
AV-Quality
Text-Consistency
AV-Consistency
Synchrony
Method
FVDJ KVDJ FADI
TV-B1+ TA-IB: CLIP) CLAPT AV-IB, CAVP1 AVHScore?
JavisScore
+T24+4VV
TempoTkn Yariv al
2024
539.8
7.2
0.084
0.205
0.137
0.787
0.122
0,103
TPoS (Joong
839.7
4.7
0.201
0.229
0.142
0.129
0.095
T2V+124
ReWaS Jeung
2024
9.4
0.123
0.280
0.110
0.794
0.104
0.079
See&Hear King et M124
7.6
0.129
0.263
0.160
0.798
0.143
0.112
FoleyCitr Zhang
9.1
0.149
0.383
0.193
0.186
0.151
T2AV
MM-Dift (Ruan al
2023
2311.9
12.2
27.5
0.080
0.014
0.181
0.079
0.119
0.783
0.109
0.070
Verserse- [Waag et
2025
194.2
0.5
8.7
0.272
0.111
0.309
0.245
0.104
0.793
0.098
0.077
JavisDiT(Ours)
204.1
1.4
7.2
0.263
0.143
0.302
0.391
0.197
0.801
0.179
0.154
Spalial Composition
Temporal Composition
Table 2: Experimental results on previous datasets. Numbers are borrowed from their re- leased papers, and our model still performs best.
0.5
6.34
0.2
TwleyC Don
6.2
Peley:
0.30
Method
Landscape
AIST+
FVD.
KVDJ FAD.
FVD.
KVD,
FAD,
0.12
MM-DiIT
26.6
49.1
0.0
342012
Mutrpie
6:10
Stuge
Suquental Servehazosa
Soc&Hear
326.2
9.2
12.7
Figure 5: Generative av-synchrony with Javis-
AV-DIT
172.7
15.4
11.2
68.8
21.0
10.2
MM-LDM
105.0
8.3
9.1
105.0
27.9
10.2
Bench's 's taxonomy. Current SOTA models still suffer from challenging scenarios.
(Our)
94.2
7.8
8.5
86.7
19.8
9.6
EXPERIMENTS
5.1 SETUP
Evaluation Datasets and Metrics. The proposed JavisBench is used as the primary benchmark for comprehensive evaluation. Multiple metries are reported: (I) audio/video quality, (2) semantic consistency against conditional texts, (3) semantic consistency and (4) spatio-temporal synchrony between videos and audios. We also evaluate on + (Li et al. 2021) and landscape (Lee et al.)
2022) for consistency, and follow prior works (Ruan al. 2023 Sun et al., 2024) report FVD, KVD, and FAD.
Compared Methods. For JavisBench, we reproduce and compare series of baselines: (1) for cascaded T2A+A2V methods Yaariv et al., 2024 Jeong et al. 2023), we use AudioLDM2 (Liu et al. 2024b) for the prepositive T2A task; (2) for cascaded T2V+ V2 22A methods (Jeong et al. 2024 Xing ct al. 2024 Zhang et al.) 2024), OpenSora (Zheng et al. 2024) is adopted to generate vidcos at currently compare with MM-Diffusion (Ruan al. 2023) and
first; and (3) for JAVG models,
UniVerse-1 Wang et al. (2025) since the others (Liu et al., 2024a Zhhoo et al.) 2025) are not currently open-sourced. For AIST+ and Landscape, we directly adopt the reported results.
Implementation Details. We collect 780K audio-text pairs for audio pertaining by 55 epochs, with 610K video-audio-text triplets train ST-Prior Estimator for 1 epoch and JavisDiT for epochs. The learning rate is lc-5 for ST-Prior Estimator and Ic-4 for DiT. The video encoder-decodee and audio encoder-decoder are taken from OpenSora (Zheng et al. (024) and AudioLDM2 (Liu al. 2024b) and kept frozen. Detailed configurations refer to Sec. C.1
5.2 MAIN RESULTS AND OBSERVATIONS
Our JavisDiT achieves superior single-Modal quality and video-audio synchrony. As shown in Tab. [ our carefully designed spatiotemporal DiT architecture demonstrates exceptional unimodal generation quality, achieving significantly superior results compared to UNet-based architectures (e.g., TempoToken (Yariv al. 2024)) and naive DIT architectures (e.g., MM-Diffusion (Ruan et al..
7

---

Published as a conference paper at ICLR 2026
group three blrds with green plumage and distinctive yellow beads are perched on branch in secene wood- land area. The bree around them have foliege. The birls are engaged in vartons activitiee, soch as grooming themsehess and looking around, whilee light chirping Cills the air.
lealless branciees
post anscabyttic urhan landscans the city filled with nubble. debris and dama- ged structurss buildings power and industrial elements are scatiered tirou- ghout, creating dystoglan ape. The sounds of debets falling, broken glass shat- teing, and occasional explosions can beard in the background.
Figure 6: JavisDiT precisely captures the visual and auditory clues from text inputs to generate faithful sounding-videos with high-quality spatio-temporal alignments. Colored texts are spatio-temporal objects (underlined) and actions. More cases shown in Sec. E.6
2023)), with remarkable FVD (204.1) and FAD (7.2) scores. Meanwhile, from the perspective of global semantic alignment, including text-consistency and video-audio consistency, our model also achieves state-of-the-arr performance, as evidenced by its TA-IB score of 0.263 and CLIP similarity score of 0.302. We attribute the gap between our JivisDiT and UniVerse-1 (Wang et al., 2025) in video-text alignment the gap of pretrained models (we use OpenSora while Uni Universe-1 takes more powerful Wan2.1 (Wan et al., (025)) ). Fig. 0 showcases some representative JAVG examples. Notably, terms of audio-video synchrony, our end-to-end model outperforms various cascaded and joint generation approaches, achieving JavisScore of 0.154, surpassing the state-of-the-ar cascaded method FoleyCrafter (Zhang et al., 2024).
To ensure a rigorous comparison, we follow the standard settings from prior works (Ruan et al. 2023 Xing et al. 2024 (un et al. (024) and train our model for 300 epochs on two closed-set datasets, including Landscape (Lee al. (022) and AIST+ (Li et al. 2021) l). As demonstrated in Tab. 2 our method consistently achieves sfate-o-thh-aae performance, with FVD of 94.2 on Landscape and FAD of 9.6 AIST++. These results further highlight the superiority of our meticulously designed DiT architecture and hierarchical spatial -emporal prior estimator.
Current models fail to simulate complex scenarios. Fig. presents Zohe al. 2024) and our Jiv across JavisBench categories, showing that existing models including ours Burs-strupp struggle with AV synchrony in complex scenarios. When the sounding video contains only single sounding object (e.g., person playing the violin alone), JavisScore is generally higher than in multi-object cases (e.g., a street performance with multiple instruments), as the latter requires identifying the correct visual-audio correspondence. Similarly, videos with multiple simultancous events (e.g., dog barking while car horn sounds) yield lower JavisScore than single-event cases (e.g.. person clapping), due to the increased challenge of modeling event timing and interactions. Sec. D.5 and Fig, A8 thoroughly highlight the limitations in handling real-world complexity.
5.3 IN-DEPTH ANALYSES AND DISCUSSIONS
To efficiently evaluate our proposed method, perform the 3rd stage (JAVG) training using a subset of 60K entries from our entire training data, and test the models on JavisBench- (with 1,000 randomly selected samples from JavisBench), For simplicity, we report three normalized scores for clear comparison from three dimensions: (1) Quality: Swvo 0.01 x STVD + SKED + 0.1 x SPAD:
(2) Consistency: SAVC = SAV.B + SCAVP + SAVTIScore: (3) Synchrony: Sws = Slavisccure-
Well-designed DIT backbone is superior. In Tab. 3 we first build vanilla baseline to replace the spatio-temporal transformer backbone by UNet (Liu et al. (024b) for the audio branch, and the generation receives pool AV-Quality at 1. After replacing the UNet backbone with STDiT modules (Zheng et al. 2024). SAvo immediately improves to 7.293, demonstrating the efficacy of the STDIT architecture. Then, respectively incorporate the HiST-Sypo estimator and the bidirectional cross-attention (BiCA) modules for video-audio information sharing. Accordingly, HiST-Sypo brings significant enhancement on AV-Consistency (1.191 I. 1.155) and AV-Synchrony
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation on the model design. The specifically developed DiT architecture and the Hyp estimator jointly contribute to single- modal quality and va-synchrony. Full evaluation results are presented in Tab, A6
Table 4: Ablation on token number and injec- tion strategies of ST-Priors. We provide more experimental results Fig. Fig.[A9]aa and Tab. A9 Full evaluation results are presented in Tab, A7
N
N₂
Injection
Quality
Consist
Synet
STDIT HiST-Sypo
BICA
Quality
Consist
Synet
0
6,581
1.157
0.133
x
9.371
1.140
0.118
1
1
CrossAti
6,909
1.188
0.137
16
CressAttn
6.127
1.191
0.150
32
32
CrossAttn
6.012
1.201
0.153
6.581
1.157
0.133
32
32
Addition
6.267
1.183
0.144
6.012
1.201
0.153
32
32
Modulate
6.190
1,191
0.145
(0.150 vs. 0.130), more than the simple BiCA module (1.157 vs. 1.155 for Svve and 0.133 vs. 0.130
for Svvs). This verifies motivation that simple channel-sharing mechanism in AV-DiT (Wang) et al. 2024b) cannot effectively build audio-video synchronization. which can be achieved by our proposed fine-grained ST-Prior guidance instead. After bridging STDiT, HiST-Sypo, and BiCA modules, our Jiv reaches the best performance for both single-modal quality (SAvQ = 6.012) and video-audio synchronization (Savc 1.201 and SAYs = 0.153), demonstrating the superiority of the well-designed DiT backbone.
Spatio-temporal prior effective and generic. In Tab. we take preliminary step investigate the number of spatio-tempora priors and the way to inject ST--riors for better video-audio synchro- nization. We first take the model without ST-CA modules (the 4th row in Tab. 3 as our baseline (the 1st row in Tab. D. and add ST-CA modules by gradually increasing the prior number from I to 32. According to the results, both single-modal quality (Sxvo) and video-audio synchrony (SAve and SAVS) consistently increase the prior number gains. Then, try to utilize the 32 spatial-temporal priors for addition (adding to video/audio latent representations like a conditional embedding) and modulation (mapping priors to scales and biases to modulate video/audio representations). Although the final performance is inferior to the utilization of cross-attention, ST-prrioss still considerably enhance all metrics (e.g., 0. 0.144/0. 145 vs. 0.133 for SAvs). Besides, further experiment and evaluate the influence of prior numbers and dimensions in Fig. A9 and verify the optimization objectives during ST-priors' estimation in Tab. A9 All the empirical results verify the effectiveness and versatility of our estimated spatio-tempora proos.
How does ST-Prior ensure synchronized video&audio? Fig. 7 illustrates the syn- chronization mechanism for our ST-priors to guide the video-audio generation pro- cess. In particular, we visualize the spatial- temporal tross-attention map from the last block JavisDiT at the last sampling step on both video and audio branches. The qualitative results Fig. show the spatial priors successfully help JavisDiT foeus the subject that would produce the sound (in this case, it is the bubbles rather than the diver that can make the sound), and the temporal priors bring nearly uniform attention scores along the timeline (as the bubbling sound continues from beginning to end). Sec, E.5 also details how spatiotemporal priors are injected and shifted over sequential events, further illustrating the well-learned cross-attention mechanism of ST-priors for synchronized video-audio generation.
Generated Video
Spatial Attention
Generated Audio
Temporal Attention
(a) Cenerated Souncing Video
Concsponetng Attimtion Map
Figure 7: Visualization cross-attention maps by spatio-temporal priors. Spatial priors successfully capture the sounding subjects (bubbles this case), and temporal priors accurately cover the whole timeline for the continuous sounding event.
JavisDiT supports variant-length sounding video generation. In the previous sections, our evaluations primarily focused on generating 4-second videos. Here, we extend the assessment to 10-second outputs and report several key metrics. The experimental results in Tab. 5 show that our JavisDiT maintains strong visual and auditory quality (FVD, FAD), semantic consistency (CLIP, CLAP, AVHScore), and audio-vide synchrony (JavisScore) when generating 10-second videos. These findings provide strong evidence that our approach effectively supports diverse audio-video generation requirements across different scenarios, durations, and resolutions.

---

Published as a conference paper at ICLR 2026
Table 5: Quantitative evaluation on variant-length generation performance.
Length
FVD4
FAD
CLIP
CLAP
AVHScoret
JavisScore
4s
241.8
7.3
0.308
0.382 0.385
0.186 0.183
0.1.53
10s
233.8
7.1
0.307
0.154
5.4 HUMAN EVALUATION
In this section, we further conduct user studies to more comprehensively evaluate the performance of joint audio-video generation through human evaluation. Specifically, we randomly sample 100 prompts from JavisBench and run Uni Verse- (Wang et al., 2025) and our JavisDiT to generate corresponding audio-vides outputs. Three volunteers are then recruited perform blind cii-tiee-oss preference judgments on three aspects: video quality, audio quality, and audio-video alignment. The averaged scores across annotators are used as the final evaluation metric.
JavisDiT vs Verser---1
Video Quality
Audio Quality
55.3%
26. 75
18.0%
A-V Alignment
38. 38.6%
56.0%
5.4%
20
40
60
80
100
Percentage (%6
Figure 8: Human evaluation on generation performance.
The human evaluation results in Fig. are largely consistent with the objective metrics reported in Tab.
(1) Our video quality is indeed slightly lower than that of Uni 1 (Wang et al. 2025). As analyzed Sec. this mainly stems from the performance gap between the backbones: our method currently relies the OpenSora-1.2 (Zheng al. 2024) backbone, which is notably weaker than the Wan-2.1 (Wan et al.) 2025) backbone used in Uni Verse-1. We expect this issue to be substantially alleviated as stronger T2V backbones become available in future versions.
(2) Our audio quality and audio-video alignment outperform UniVerse- This is because UniVerse-1 adopts a relatively complex architecture to bridge two separately pretrained branches, and the resulting representation gap weakens cross-modal interaction. In contrast, our JavisDiT design offers a more streamlined and effective mechanism for enabling tight oudio-video coupling.
CONCLUSION
This paper presents JavisDīT, novel Joint Audio-Video Diffusion Transformer that simultancously generates high-quality audio and video content with precise synchronization. We introduce the HiST-Sypo Estimator, fine-grained spatio-temporal alignment module that extracts global and fine-grained priors to guide the synchronization between audio and video. We also propose the JavisBench dataset, comprising 10, high-quality text-captioned sounding videos with diverse scenes and real-world complexity, addressing limitations in current benchmarks. In addition, introduce a temporal-an aware semantic aligmment mechanism to better evaluate JAVG systems on complex content. Experimental results show that JavisDiT outperforms existing approaches in both content generation and synchronization, establishing a new benchmark for JAVG tasks. Potential limitations and future work discussed in Sec. A.5
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXiv:2303.08874. 2023.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020b.
Xiaowei Chi, Yatian Wang, Aosong Cheng, Pengjun Fang, Zeyue Tian, Yingqing He, Zhaoyang Liu, Xinggun Qi. Jiahao Pan, Rongyu Zhang, al. Mmtrail: multimodal trailer video dataset with language and music descriptions. arXiv preprint arXiv:2407.20962, 2024.
Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng Hc, Junyang Lin, et al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759, 2024.
Marco Comunità, Riccardo Gramaccioni, Emilian Postolache, Emanuele Rodolà, Danilo Com- miniello, and Joshua D Reiss. Syncfusion: Multimodal cnset-synchromized video-to-audio foley synthesis. ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 936-99 940. IEEE, 2024.
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 736-740. IEEE, 2020.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference on machine learning, 2024.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36. 2024.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP). IEEE, 2017a.
Jort Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEBE, 2017b.
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15190-15190, 2023.
Akio Hayakawa, Masato Ishii, Takashi Shibuya, and Yuki Mitsufuji. Discriminator-guideed cooperative diffusion for joint audio and video generation. arXiv preprint arXiv:2405. 2024.
Jonathan Ho. Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in
neural information processing systems, 33:6840-6851. 2020.
Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruigi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint 2r.iv:2210.02303 2022.
11

---

Published as a conference paper at ICLR 2026
Yuchen Hu, Yu Gu, Chenxing Li, Rilin Chen, and Dong Yu. Video-to-auue -udii generation with fine-grained temporal semantics. arXiv preprint arXiv:2409 1et.29999.4479,, 2024.
Rongjie Huang, Jiawei Huang, Dongchao Yang. Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Make-an-audio: Text-to-audio generation with prompt-enhanced diffusion models. In International Conference Machine Learning, pp. 13916-13933 PMLR, 2023.
Vladimir lashin, Weidi Xie, Esa Rahtu, and Andrew Zisserman. Synchformer: Efficient synchroniza- tion from sparse cues. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5325-5329, IEEE, 2024.
Masato Ishii, Akio Hayakawa, Takashi Shibuya, and Yuki Mitsufuji. A simple but strong baseline for sounding video generation: Effective adaptation of audio and video diffusion models for joint generation. arXiv preprint arXiv:2409.17550 2024.
Yujin Jeong, Wonjeong Ryoo, Seunghyun Lee. Dabin Sco, Wonmin Byeon, Sangpil Kim, and Jinkyu Kim. The power of sound (tpos): Audio reactive video generation with stable diffusion. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 7822-7832, 2023.
Yujin Jeong, Yunji Kim, Sanghyuk Chun, and Jiyoung Lee. Read, watch and scream! sound generation from text and video. arXiv preprint vrXiv:240..0555/ 2024.
Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 119-132, 2019.
Seung Hyun Lee, Gyeongrok Oh, Wonmin Byeon, Chanyoung Kim, Won Jeong Ryoo, Sang Ho Yoon, Hyunjun Cho, Jihyun Bae, Jinkyu Kim, and Sangpil Kim. Sound gounde semantic video generation. In European Conference on Computer Vision, pp. 34-50. Springer, 2022.
Ruilong Li, Shan Yang. David Ross, and Angjoo Kanazawa. Ai choreographer: Music conditioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13401 11000-134122 13412, 2021.
Minghui Liao, Zhisheng Zou, Zhaoyi Wan, Cong Yao, and Xiang Bai. Real-time scene text detection with differentiable binarization and adaptive scale fusion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Yaron Lipman, Ricky TQ Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
Haohe Liu, Zehua Chen, Yi Yuan, Xinhao Mei, Xubo Liu, Danilo Mandic, Wenwu Wang, and Mark D Plumbley. Audioldm: Text-to-audio generation with latent diffusion models. arXiv preprint arXiv:2301.12500 2023a.
Haohe Liu, Gael Le Lan, Xinhao Mei, Zhaoheng Ni, Anurag Kumar, Varun Nagaraja, Wenwu Wang, Mark D Plumbley, Yangyang Shi, and Vikas Chandra. Syncflow: Toward temporally aligned joint audio-video generation from text. arXiv preprint arXiv:2412.15220 2024a.
Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. Audioldm 2: Learning holistic audio generation with self-super pretraining. JEEE/ACM Transactions on Audio, Speech, and Language Processing. 2024b.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, In European Conference on Computer Vision, pp. 38-55. Springer, 2025.
12

---

Published as a conference paper at ICLR 2026
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations (ICLR), 2023b.
Xubo Liu, Qiuqiang Kong, Yan Zhao, Haohe Liu, Yi Yuan, Yuzhuo Liu, Rui Xia, Yuxuan Wang. Mark D Plumbley, and Wenwu Wang. Separate anything you describe. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024c.
Jiayi Lyu, Xing Lan, Guohong Hu, Hanyu Jiang, Wei Gan, and Jian Xue. Etau: Towards emo- tional talking head generation via facial action unit. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6, 2024. doi: 10.1 109/CCME57554 2024, 10687525.
Jiyy Lyu, Xing Lan, Guohong Hu, Hanyu Jiang. Wei Gan, Jinbao Wang, and Jian Xue. Multimodal emotional talking face generation based on action units. IEEE Transactions Circuits and Systems for Video Technology, 35(5):0026
2025. doi: 10. (0.119//CSVV122243323333
Yuxin Mao, Xuyang Shen, Jing Zhang, Zhen Qin, Jinxing Zhou, Mochu Xiang, Yiran Zhong, and Yuchao Dai. Tavgbench: Benchmarking text to audible-video generation. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 6607-6616, 2024.
Irene Martín-Morató and Annamaria Mesaros. What is the ground truth? reliability of mulli--nnooatorr data for audio tagging. In /02/ 29th European Signal Processing Conference (EUSIPCO), pp- 76-80. IEEE, 2021.
Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions Audio, Speech, and Language Processing, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the JEEE/CVF International Conference on Computer Vision, pp. $195-4205 2023.
Karol J Piezak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 1015-1018, 2015.
Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih- Yao Ma, Ching- Yao Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint arX1v:2410.13720,, 2024.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21(140):1-61 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125. 1(2):3 2022.
Nikhila Ravi, Valentin Gabeur, Juan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.000 007144 2024.
Yong Ren, Chenxing Li, Manjie Xu, Wei Liang, Yu Gu, Rilin Chen, and Dong Yu. Sta-v2a: Video- tu-audio generation with semantic and temporal alignment. arXiv preprint arXi::2409.8800 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer on computer vision and pattern recognition, pp. 11684-10695, 2022.
Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 00219-10228. 2023.
13

---

Published as a conference paper at ICLR 2026
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems, (1.36479-36494 2022.
Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pp. 1041-1044, 2014.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2/1 2021.
Xuyang Shen, Dong Li, Jinxing Zhou, Zhen Qin, Bowen He, Xiaodong Han, Aixuan Li, Yuchao Dai, Lingpeng Kong, Meng Wang, et al. Fine-grained audible video description. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp- 10585-10596 2023.
Bob L Sturm. The gtzan dataset: Its contents, its faults, their effects on evaluation, and its future use. arXiv preprint arXiv: 1306. 1461, 2013.
Mingzhen Sun, Weining Wang, Yanyuan Qiao, Jiahui Sun, Zihan Qin, Longteng Guo, Xinxin Zhu, and Jing Liu. Mm-ldm: Multi-modal latent diffusion model for sounding video generation. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 10853-10861, 2024.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Jason Wei, Xuezhi Wang. Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, et al. U12: Unifying language learning paradigms. arXiv preprint (xXiv:2205005133 2022.
Team Wan, Ang Wang. Baole Ai, Bin Wen, Chaojie Mao, Chen-Wai Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, et al. Wan: Open and advanced large-scale video generative models. arXiv preprint ar arXiv:2503.20314 2025.
Ao Wang, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. Repvit: Revisiting mobile cnn from vit perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15909-15922 2024a.
Duomin Wang, Wei Zuo, Aojie Li, Ling-Hau Hao Chen, Xinyao Liao, Deyu Zhou, Zixin Yin, Xili Dai, Daxin Jiang, and Gang Yu. Universe-1: 1: Unified audio-video generation via stitching of experts. arXiv preprint arXiv:2509.066 06155, 2025.
Kai Wang, Shijian Deng, Jing Shi, Dimitrios Hatzinakos, and Yapeng Tian. Av-dit: Efficient audio- visual diffusion transformer for joint audio and video generation. arXiv preprint arXiv:2406.07686, 2024b.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing rision-languae model's 's perception of the world at any resolution. arXiv preprint 24Xiv:2409. 12/91, 2024c.
Yaohui Wang, Xinyuan Chen, Xin Ma, Shangchen Zhou, Ziqi Huang, Yi Wang. Ceyuan Yang, Yinan He, Jiashuo Yu, Peiqing Yang, et al. Lavie: High-quality video generation with cascaded latent diffusion models. International Journal of Computer Vision, pp. 1-20, 2024d.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal 11m. In Forty-first International Conference on Machine Learning.
Zhifeng Xie, Shengye Yu, Qile He, and Mengtian Li. Sonicvisionlm: Playing sound with vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26866-26875, 2024.
Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang. and Qifeng Chen. Seeing and hearing: Open- domain visual-audi generation with diffusion latent aligners. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 7151-7161, 2024.
14

---

Published as a conference paper at ICLR 2026
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024a.
Zhuoyi Yang. Jiayan Teng. Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang. Wenyi Hong, Xiaohan Zhang, Guanyu Feng, al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408. 06072, 2024b.
Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi. Diverse and aligned audio-to-vicee generation via text-to-video model adaptation. In Proceedings of the AAAI Confer- ence on Artificial Intelligence, volume 38, pp. 6639-6647, 2024.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng. Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:3310.05737, 2023.
Lin Zhang, Shentong Mo, Yijing Zhang, and Pedro Morgado. Audio-synchronized visual animation. In European Conference on Computer Vision, pp. 1-18. Springer, 2025.
Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. Foleycrafter: Bring silent videos life with lifelike and synchronized sounds. arXiv preprint arXiv:2407.01494 2024.
Lei Zhao, Linfeng Feng, Dongxu Ge. Fangqiu Yi, Chi Zhang, Xiao-Lei Zhang, and Xuelong Li. Uni- form: unified diffusion transformer for audio-video generation. arXiv preprint arXiv22502.08897, 2025.
Zangwei Zheng, Xiangyu Peng, Tianji Yang, Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024. URI. / tigituuuubb hub. com hww/hpca twch/Dpen-hhheeeeeeeeee
Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, and Yiran Zhong, Audio-visual segmentation. In European Conference on Computer Vision, 2022.
15

---

Published as a conference paper at ICLR 2026
A DISCUSSIONS
A.1 ETHICAL STATEMENT
In this work, we construct our JavisBench from publicly available academic datasets well as YouTube videos. To ensure ethical compliance, we strictly adhere to YouTube's Term of Service and licensing policies. Specifically:
1. Privacy Protection: We have taken measures to remove any personally identifiable or privacy- sensitive information from the collected data. No private, confidential, user-specific metadata has been retained.
2. Copyright Compliance: All data collection respects the original content licenses. We only utilize publicly accessible videos that are either explicitly licensed for research use or fall under fair-use considerations. No copyrighted content is redistributed modified in violation of licensing terms.
3. Responsible Data Release: Any potential dataset release will fully comply with YouTube's data policies and ethical guidelines. We will first release the self-curated caption data to support the evaluation of all metrics except the AV-Quality terms Tab. [ and then ensure that shared counding-videst are either appropriately anonymized or restricted in accordance with relevant regulations and platform policies.
By implementing these measures, we strive to maintain high ethical standards in data collection, use. and dissemination.
A.2 SOCIETAL IMPACT STATEMENT
This work proposes JivisDīT, unified model for synchronized audio-video generation from open- ended prompts. While primarily intended as a contribution to foundational multimodal generation research, it has both positive applications and potential societal risks.
Positive Impact. JavisDiT may support creative industries by enabling efficient generation of audio-video content. Addressing the challenge of audio-video alignment has practical implications for animation, video conferencing, television broadcasting, and video editing-domains where synchronization traditionally achieved through offline computation or extensive manual post- processing.
Potential Risks. As with other generative models, JavisDiT may be misused to produce highly realistic synthetic media for malicious purposes, including misinformation, impersonation, and manipulation. Its ability to generate temporally coherent sounding videos may increase the realism and persuasiveness of deepfakes. Specifically, fabricated audio video clips could be used to spread false information or simulate events that never occurred, posing risks to public trust, political discourse, and media integrity.
mitigate these concerns, we recommend responsible release practices, including usage gating and dataset transparency. Future work may explore watermarking and detection tools to distinguish synthetic output.
Mitigation Strategy.
A.3 REPRODUCIBILITY STATEMENT
We provide detailed descriptions of model design, training, and evaluation in both the main paper and the appendix. Furthermore, all code, pretrained checkpoints, and processed datasets will be publicly released to ensure full reproducibility of our results.
A.4 LLM USAGE STATEMENT
Large Language Models (LLMs) were used solely as writing assistants, including tasks such as language polishing and presentation refinement. They were not involved in the conception of core ideas or designs.
16

---

Published as a conference paper at ICLR 2026
Table Al: Latency analysis for 240P, sounding-vides generation on H100 with 30 sampling steps.
Video Branch
Audio Branch
ST-Prior Modulation
AV-Interaction
Overall
13s
2s
13s
2s
30s
A.5 POTENTIAL LIMITATION AND FUTURE WORK
Despite the strong performance in joint audio-video generation, our JavisDiT has several limitations that present opportunities for future research:
1. Scalability of Training Data: Our model was trained on 0.6M text-video- triplets, which, while substantial, is still limited compared to the scale of some large usioonlanguage models (e.g., OpenSora (Zheng et al., 2024) takes over 60M data to build foundation mode for video generation). Expanding the dataset with more diverse and higher-quality real-world audio-video samples could further enhance the model's ability to generalize across different domains and fine-grained synchronization patterns.
2. Synchronization Evaluation Metries: While we introduce JavisScore for evaluating audio- video synchronization, its current accuracy of 75% suggests room for improvement. More robust synchronization metrics potentill incorporating perceptual alignment assessments or human-in-the-loop, evaluation- further refine synchronization quality measurement in JAVG research.
3. Efficiency and Computation Overhead: JavisDiT employs a Diffusion Transformer for high-quality generation, but diffusion-based models tend to be computationally intensive, as analyzed in Tab, AT While our approach achieves state-of-the-art results, generation speed and siadddin-vidoooooo efficiency remain challenges (e.g., generating a 2-second sounding video 720P/24fps/ 1/kkfp///6HK 6kHz takes 6 minutes on one H100 GPU). Exploring accelerated sampling strategies or hardware optimization could improve efficiency.
4. Benchmarking Across Resolutions and Durations: Our evaluations primarily focus fixed resolution (240P) and duration (4s) setting in JavisBench. However, real-world applications may require generation at higher resolutions (e.g., 1080P) and longer durations. Conducting benchmark tests across multiple settings would provide a more comprehensive understanding of current models' strengths and limitations.
By addressing these limitations, future iterations of JavisDiT could further enhance scalability, synchrony, efficiency, and adaptability, pushing the boundaries of joint audio-video generation.
B RELATED WORK
In the field of AIGC, multimodal generation has become a key topic, encompassing text-to-image
image
(Rombach et al.) 2022; Ramesh et al. 2022. Saharia et al.) 2022), text-to-video (Ho et al.
2022
Wang et al. 2024d) Yu et al. 2023). and text-to-audio (Liu et al.| 2023a 2024b Huang et al.
2023)
generation, etc. Among these, sounding video generation (Ruan et al. | 2023 Xing al. 20224
Wang
ct al. 2024b] Sun et al. 2024 Liu et al.] (024a) is drawing increasing attention due to its strong alig alignment with real-world applications. To generate synchronized audio and video, early works decomposed the task into two cascaded subtasks. a) generating video from text, then adding audio
2024 Hu et al. 2024]; or b) generating audio first, then synchronizing video Yariv et al. 2024 Jeong et al. 2023 Zhang et al. 2025). As an instance, MovieGen (Polyak et al. 2024) achieves movie-grade generation quality using this cascaded approach. From methodological perspective, the community focus has shifted from UNet- models (Rombach et al., [022] Liu al.) 2024b) to DiT-based methods (Zheng et al.
(Comunità et al. 2024 Jeong et al., 2024 Zhhng al. [2024] Ren
2024
Polyak et al. 2024) to achieve more state-of-the-ar performance.
Another line of work considers the limitations pipeline approaches, such as error propagation and reliability issues, by focusing on end-to-end Joint Audio Video Generation (JAVG). This paradigm aims to improve generation quality by modeling audio-video synchronization, including semantic consistency and temporal alignment, and has received growing attention (Ruan et al. 2023 Hayakawa et al. 2024 Ishii et al. 2024). For example, Ruan et al.] (2023) introduce a diffusion block that enables cross modal interaction during denoising. Xing al. (2024) propose using ImageBind to align the semantic representations of the two modalities, [Sun et al. (2024) introduce hierarchical VAE, mapping audio and video to shared semantic space while reducing their dimensionality.
17

---

Published as a conference paper at ICLR 2026
Video
Text Prompt
Audio
Table A2: Detailed settings for three-stage training.
buckyand picket green grass....
Setting
Stage-1
Stage-2
Stage-3
Video Encoder
Encoder
Audio Encoder
trainable params
1.11B
29.3M
923.8M
learning rate
lc-4
lc-5
lc-4
Spatie-tamporl Self-attetton
Spario-temporal Self-atention
warm-up steps
1000
1000
1000
Iunaliiin
Eand
weight decay
0.0
0.0
decay
0.99
0.99
Course-gaaied
Courso-grained Croes astetioon
Cross actention
Muntalcing
dropout rate
0.0
0.0
training samples
788K
611K
611K
FeedForward
Tennlining
FoodForward
batch size
dynamic 55
dynamic
dynamic
epoch training objective
2
Audion
rect. flow
contrastive rect. flow
Figure A1: Audio Pretraining. Parameters are initialized from the video branch.
GPU days (H100)
64
256
AV-DiT (Wang et al. 2024b) employs single DiT model to generate both video and audio modalities simultaneously, improving efficiency. Uniform (Zhao et al., 2025) also takes a single DiT for JAVG, by simply concatenating the video and audio latent tokens during the diffusion process, without any explicit synchronization guidance. SyncFlow (Liu et al.) 2024a) utilizes STDiT (Zheng et al. (024) blocks to enhance the video generation quality, with temporal adapter to guide the audio generation process. Unfortunately, current JAVG methods either lack strong backbone for audio and video generation insufficiently model audio-video synchronization, resulting in suboptimal generation quality. To address these issues, we construct a novel DiT-based JAVG model, further enhanced by modeling hierarchical spatial-tempp nential-emmpoal synchronized prior features. Moreover, this paper aims to further advance JAVG by providing a more comprehensive, robust, and challenging benchmark.
C IMPLEMENTATION DETAILS
C.I JAVISDIT MODEL CONFIGURATION
Model Architecture. As Fig. 2 illustrates, JavisDiT consists of two branches for video and audio generation, each comprising N 28 DiT blocks. Within each DiT block, the latent video and audio representations are processed through several modules, where all attention modules utilize 16 attention heads with hidden size of 1152. The intermediate dimension of the FFN is 4x the hidden size. Each attention and FFN module is preceded by a LayerNorm layer. In practical implementation, every attention module is followed by an FFN module to further enhance feature modeling. The vidco/audio latent is sequentially processed with: Spatial-SelfAAtn CrossAtm Spatial-CrossAtre CrossAtm Bi-CorssAttn - FFN Temporal-Sede Sellorft - CrossAtt Temporal-Creet CrrsssAmm Bi-CorssAtm FFN for N 28 times, resulting in our JivisDiT with 3. parameters in total.
In particular, we adopt neatial-temporll (self- or cross-) attention modules primarily to reduce computational cost. For video tokens of size (T x H x W), the computational complexity of full self-attention is ((TH)?)) In contrast, ((T(HW)==2'0 (HW)) = O(THW- (T' + HW)), which substantially lowers both memory consumption and computational overhead.
attention reduces the complexity to
Training Strategy. As Sec. 3.3 states, we carefully design a three-stage training strategy to achieve high-quality single-modal generation and ensure fine- inee-griieed spatio-temporal synchrony on generated videos and audios. In particular, the video branch of JavisDiT (including the ST-SelfAttn module, the Coarse-Graind CrossAttn module, and the FFN module) initialized from OpenSora (Zheng al. 2024) and frozen during the whole training stages. We the weights of the video branch to initialize the audio branch for better convergence (see Fig. AI The detailed configuration for the three-stage training strategy is displayed in Tab. A2
For audio mnddoopretrainin (stage-1), the audio-caption pairs come from various public datasets, including AudioSet (Gemmeke al.) 2017b), AudioCaps (Kim al. 2019). VGGSound (Chen et al. 2020a),, WavCaps (Mei et 2024). Clotho (Drossos et al. (020)) ESC50 (Piczak 2015), GTZAN (Sturm) 2013), MACS (Martin-Morató & Mesaros, 2021) and
Training Data Curation.
18

---

Published as a conference paper at ICLR 2026
Figure A2: T-SNE visualization of prompt distribution of the training dataset.
UrbanSound8K (Salamon et al. 2014). They contribute to a total of 788K training entries. For the ST-Prior estimator and the final JavisDiT training (stage-2&3) 2),,, the data source comes from two newly-proposed sounding video datasets: MMTrail (Chi et al. (024) and TAVGBench (Mao et al.) 2024). MMTrail provides slightly more caption annotations (2M) than TAVGBench (1.7M) but has lower quality. Due to the resource limit, we collect a part of the original sounding-vides from YouTube and filtered out around 80% human-talkien
videos with the FunASR tool. We finally triplets from MMTrail and 475K from TAVGBench, resulting in 61 K data for training in total. Fig. A2 visualizes the prompt distribution, which further demonstrates the diversity of our collected training dataset. The data construction for ST-Prior estimator's contrastive learning discussed in Sec. C.2.4
collect 136K high-resolution video-audio-caplie
Training Objective and Inference. The ST-Prior Estimator is trained with contrastive learning objectives, which will be detailed in Sec. C.2 For the DiT model, we rectified flow (Liu et al. 2023b) as the denoising scheduler for better performance (Zheng et al. 2024 Polyak et al. 2024 The inference sample step 30. with classifier-free guidance at 7.0. The video and audio latent are concurrently sampled at each inference step. The video encoder-decoder comes from OpenSora (Zheng al., (024), and the audio encoder-decoder comes from AudioLDM2 (Liu et al..
2024b). Both of them are frozen during training.
Detailed Evaluation Setup. In Sec. Sec.5lin our manuscript, we conduct a comprehensive evaluation across multiple video and/or audio generation models on two settings: (1) our proposed Javis- Bench benchmark and (2) previously used AIST+ (Li et al. 202I] and Landscape (Lee ct al. 2022) datasets. For JavisBench, generative models required to generate videos and 240p and 24fps. with audios at 16kHz sample rate. The duration is 4 seconds. For AIST+ (Li et al., 2021). we follow
the literature (Ruan al. 2023 Sun al. 2024 to generate 2-second clips at the visual resolution
of 256 x 256, with 240p on landscape (Lee et al. 2022).
C.2 ST-PRIOR ESTIMATOR CONFIGURATION
C.2.1 OVERVIEW
Since ImageBind (Girdhar et al. 2023) has built consistent semantic space across multiple modali- ties, ImageBind's text encoder to extract the potential apatio-temporal prior from the input text caption. Formally, for a given input text s, we leverage the 77 hidden states from the last layer of ImageBind's text encoder, and learn N₂ spatial tokens ps and N₂ temporal tokens Ps from 4-yyer transformer block P. Notably, since the same event may occur at different locations and timestamps in different video-audio generations, the same text s can also produce different ST-Priors
19

---

Published as a conference paper at ICLR 2026
Video
[T..Sp.D)
To Sg. DJ
Interpolation
Interpolation
Video Encoder
Audio Encoder
PT D]
[..
[T.
IT.S P2]
VA-Fuser
Bidirectional Cross Attention
Bidirectional Crose
[1,S,D]
[1.S,D]
[111,]]
D]
PA-Sparial
VA-Temporal
MLP
MLP
[S
[T.D]
Figure A3: Architecture of VA-Fuser to encode spatio-temporal embeddings.
(p&, Pr). We adopt sampling strategy akin VAE, where our ST-Prior Estimator outputs the mean and variance of a Gaussian distribution, from which we sample plausible (P.. Pt) to describe specific apatio-tempont events. This can be formalized as: (P.. Pt) (s c), where refers the normalized Gaussian distribution.
As Fig. states, the ST-Prior Estimator Pg is trained with contrastive learning. Specifically, we take the text prompts and estimated prior (Ps: P:) as an anchor, and treat the corresponding synchronous video-audio (v, a) pairs from training datasets as positive samples. Then, we either synthesize an asynchronous video v audio a and take the original or to form negative pairs.
Formally, we also extract Ng/N; positive or negative spatial/temporal embeddings from synchronous or asynchronous video-audio pairs:
(e e, Ev a or Ev a).
Sec. C.2.2 displays more details.
The training objective is to make the priors (p., P:) closer to the synchronous embeddings (et e," while pushing them further from the asynchronous ee ). thereby equipping the priors (p... P:) with fine-grained conditioning capabilities. Several contrastive losses with the Kullback-Lebleer (KL) divergence loss are utilized to optimize the spatial and temporal priors (see Sec, C.2.3):
Sprior Leontrast (P., c c,) Lu(P.) + Leontras (Pr, ot, c7) + Lu(P:))
(A1)
The details for negative sample construction are provided in Sec. C.2.4
C.2.2 SPATIAL-TEMPORT VA-ENCODING
For given video clip with the corresponding audio, first OpenSora's 's (Zheng et al. 2024) and AudioLDM2's (Liu et al. (022bb VAE-encoders to respectively map the raw data to the feature space, and derive lightweight VA-Fuser module extract the spatio-temporal embeddings from the video-audio pair. As shown in Fig. A3 the raw video and reshaped to video embedding v (V) E x.TxXS.DD, where the downsampling rate T
(H will be encoded
for (T' is (4, 8, 8), and the output channel D 4. Similarly, the raw audio will be first transformed to the MelSpectrogram A E R RxxMxx and then encoded to audio embedding (A) IR" T.Txxx x We use a linear layer to unify video and audio channels into the hidden size D = 512.
Then, we align the video and audio embeddings with the pre-defined spatial-temporal token numbers via linear interpolation. Specifically, we will obtain video-spatial embedding v, RT.xSXD a video-temporal embedding V, E xT' a audio-spatial embedding a, E R XSXD and RTxS.xD. xD Subsequently,
audio-temporal embedding
apply bidirectional attention (see Fig. 2 d)) for the spatial va-embedding Bareding--aiiraaa (v., a.) and the temporal va- aa-embedding--ee After averaging the temporal dimension (T., Ta) and merging the video-audio embeddings, a 2-layer MLP module to project to the desired video-audio spatial embedding e, e R SxD With similar operations,
(ve, a.).
obtain the iudoo-audio temporal embedding e; E RD'D
20

---

Published as a conference paper at ICLR 2026
As defined in Sec. the extracted spatial embedding e, should determine the occurrence of specific event in a region of the video frame, with the corresponding appearance of matching frequency components in the audio spectrogram. On the other hand, the extracted temporal embedding e, should identify the onset or termination of an event at a specific frame or timestamp in the video, with the corresponding start stop of the response in the audio. We achieve these goals by dual contrastive learning objective.
C.2.3 CONTRASTIVE TRAINING LOSSES
Our core objective is to align the spatial/tempora priors (ps. P.) (extracted from text) more closely with the positive spatial/temporal embeddings (e) ., derived from synchronized video-audio pairs, while pushing them away from the embeddings (e, e, of negative samples (asynchronous video-audio pairs). This ensures that the priors (Ps: P:) provide the fine-grained spatio-temporal condition required for synchronized video-audio generation.
To achieve this, we designed four contrastive loss functions. For simplicity, we omit the subscripts of 8, / this part, as we apply the same loss functions on both spatial and temporal priors/embeddings.
1. Token-level hinge loss: Ltoken ep.eet,ee ,-) |1.0- D sim(p, et) +1.0 + sim(p. eT>], where
sim refers to the cosine similarity.
2. Auxiliary discriminative loss:
LBCE(De(D, e)),
(DC ).).)), where DD a I-layer attention module parameterized with that use the [CLS]] token to gather the information in p and
3. VA-embedding discrepancy loss: Lvad(e*, ,e) 1.0 sim(e*, ,e)l, which aims to enlarge the discrepancy of positive va-embedding e and negative e
4. 22-reularization loss: Greg (p, et) = ||p - et l12 which directly pushes the prior towards
the positive (synchronous) va-embedding e+
Overall, the contrastive learning loss is combined with:
Ccommus (p, et =Lokken (p, et, eet,e"))) + Laise (p,, et, ,e)) + Luad(e", , + Lreg (p,
(A2)
Specifically, we take the text prompts and estimated prior (p., Pr) as an anchor, and treat the corresponding synchronous video-audio (v, a) pairs from training datasets as positive samples. Then, we either synthesize an asynchronous video v audio a and take the original negative pairs. It can be formulated as (e e, = a). (e, e, (v, a or (v)..)). The VA- Encoder E detailed Sec. [.2.2] and the negative sample construction introduced in We also discuss the efficacy of each loss function in Sec. E.4
form
Sec. C.2.4
C.2.4 NEGATIVE SAMPLE CONSTRUCTION
Given a synchronized video-audio pair from the training set, we design two approaches to syn- thesize easy and hard asynchronous videos/audios for optimization. (I) For easy negatives, take AudioLDM2 (Liu et al.) 2024b) to generate arbitrary audios from the text without referencing the corresponding video, which naturally results in asynchronous video-audio pairs. (2) For hard negatives, various augmentation strategies are employed to add, remove, or modify elements in either the original video or audio from GT--pairs, creating more fine-grain fine-grained spatio-temporally asynchronous pairs. As detailed below, these methods further enhance the discriminative power of the ST-Prior, and Fig. A4 and Fig. A5 showcase some representative augmentation examples.
Video Spatial Augmentation
Random Masking (Fig. A4 A)). Videos are divided into 6 6 grids, with a mask ratio p uniformly sampled from (0.2, 0.8)). A proportion of the grids is randomly masked.
Adding Subject Trajectories (Fig B)) The trajectories from the SA-V dataset (Ravi et al. 2024) overlaid simulate new sound-producing objects. Trajectories preprocessed using RepVIT-SAM (Wang et al. 2024a) for interpolation from 6fps to 24fps, retaining objects with an average size >> 32 x 32 pixels. During augmentation, random trajectory is selected from the pool and added to the video.
Video Temporal Augmentation
21

---

Published as a conference paper at ICLR 2026
Original Video
Video Spatial uumentation
(A) Random grid masking of video.
(B) Addition other segmentation trajectories.
Video Temporal Augmentation
(C) Video temporal shifting.
(D) Video pausing.
Figure A4: Video augmentation for spatial and temporal negative samples.
Video Temporal Shifting (Fig. A4 C)). The video is cyclically rearranged from random starting
point to create temporal misalignment.
Video Pausing (Fig, A4()). A random frame is duplicated for at least 0.5 seconds at chosen point, pausing the video and disrupting temporal synchronization.
Audio Spatial Augmentation
All audio augmentations (except temporal shifting) focus on cound-producing intervals, avoiding silent noisy segments. We use QwenPlusAPP to extract potential sound- grodd-prouucingg objects from audio descriptions, and utilize AudioSep (Liu er al., 2024c) to separate sound sources and labels intervals using RMS. Separation results precomputed and stored metadata, guiding target augmentations.
Audio Source Addition (Fig. A5 A)). Additional audio from other dataset sources is mixed with the original one, causing spatial content misalignment between audio and video. Audio Source Removal (Fig. B)). separated sound source from the original audio is randomly removed to induce more spatial misalignments.
22

---

Published as a conference paper at ICLR 2026
AriginalAudio
Audio Temaoral uguestation
0.35
8.15
0.15
0.10
5.19
10
0.65
5:05
0.05
b.03
-5.55
0.05
Audio Spatial Augmentation.
(D) Audio temporal shifting. Left: original audin; Right: augmentted audio.
0.05
6.05
0.65
0.00
6.00
-0.55
-6.05
-0.65
-0.10
0.15
0.35
Addition audio
Insertion silent segments. Left original audio; Right: augmented audio.
0.15
5.15
0.15
0.10
5.19
6.18
0.65
5.85
6.08
-0.15
Canelinc's
timcinch
lindiiess
Removal audio sourrees.
(F) Insertion of repeated segments. Left: original audio: Right: angmeented audio.
0.15
0.15
6.15
0.10
septinad
-0.15
-0.15
-0.15
Smalliee'
tinelias'
djjustmenn of audio volume.
(G) A djustmmnnt of audio speed. Lelt: original audio: Right: augmented audio.
Figure A5: Audio augmentation for spatial and temporal negative samples.
Audio Volume Adjustment (Fig. A5 C)). The target audio's volume is modified by randomly
selected transformation (e.g., cosine, sine, linear increase/decrease) to adjust its amplitude.
Audio Temporal Augmentation
Audio Temporal Shifting (Fig. A5 D)). Audio is shifted by extracting new interval from the
original, creating temporal misalignment.
Silent Segment Insertion (Fig. AS Silent segments are inserted into the target interval, disrupting synchronization with the video timeline in subsequent frames.
Repeated Segment Insertion (Fig. F)). Target segments are repeated after a random transfor- mation interval, making another type of timeline misalignment.
Audio Speed Adjustment (Fig. G)). Playback speed within the target interval is altered to 0.5x or 2x, causing another type of timeline desynchronization.
D MORE DETAILS ON JAVISBENCH
D.I MOTIVATION
strong generative model must produce diverse video content and audio types while ensuring fine-gre aine-granedd spatio-temporal synchromization. However, have noticed that current benchmarks and metrics lack sufficient diversity and robustness for comprehensive evaluation. For instance, the commonly used AIST+ (Li 2021) and Landscape (Lee et al. 2022) benchmarks focus
23

---

Published as a conference paper at ICLR 2026
only on limited scenarios (human dancing for AIST+ and natural scenarios for landscape) with merely 20-100 test samples. On the other hand, although TAVGBench (Mao et al., 2024) provides 3,000 test samples (but not released yet), it lacks detailed analysis of different scenarios in practical audio-video generations. While larger-scale datasets such as VGGSound (Chen et al., 2020a) and AudioSet (Gemmeke al. (017aa) provide millions of video-audio pairs for training and testing, they are typically limited to a singlc-event taxonomy (e.g., human voice, engine) and do not reflect multi-event dynamics within single video clip. Consequently, they are not suitable for evaluating spatial-temp temporal alignment in complex real-world environments. Moreover, all videos in VGGSound are captured in the wild, resulting in lack of in-house scenes. Both of the two datasets fall short in scene diversity, underrepresenting diverse categories such as 2D/3D animations and industrial settings, which are crucial for evaluating generative models under varied and challenging conditions. In addition, the video-audio synchronization metric AV-Align (Yariv et al. (2024) cannot produce accurate evaluation on complicated scenarios, e.g., with multiple sounding events (Ishii et al. 2024
Mao et al. 2024).
We are therefore motivated to propose a more challenging benchmark to evaluate generative models from various dimensions larger data scale in Sec. 4.T with particular focus on spatio-temporal synchronization. We also introduce a robust metric to quantify the fine-grained apatio-tempon alignment between generated video and audio pairs in Sec. 4.2 Here we provide more details of the benchmark construction and metric verification.
D.2 TAXONOMY
To comprehensively evaluate the capabilities of joint video-audio generation models, we designed five evaluation dimensions from different aspects: (1) Event Scenario, (2) Video Style, (3) Sound Type, (4) Spatial Subjects, and (5) Temporal Composition. Leveraging GPT-4 (Achiam et al. 2023), we enumerated 3-5 primary categories under each aspect, contributing to a hierarchical categorization system with 5 dimensions and 19 categories. Tab. A3 presents the detailed definitions and clarifications.
D.3 DATA CURATION
D.3.1 COLLECTION
We collected data from two sources construct the benchmark. First, we incorporate existing datasets' test sets, including those in the JAVG domain like Landscape (Lee et al.) 2022) and AIST++ (Li al. 2021) (despite their limited coverage) and others from counding-video understanding tasks such as FAVDBench (Shen et al. 2023) and AVSBench (Zhou et al. 2022). Second, we expanded the benchmark by crawling videos uploaded YouTube between June 2024 and December 2024 to prevent data leakage in previous methods' training scope (Mao et al., 2024). Using the previously defined categorization system, we prompt GPT4 to generate potential keywords for specific categories, enabling targeted video collection and avoiding indiscriminate gathering of homogeneous data, which significantly improves curation efficiency. This stage yields around 30K sounding video candidates.
D.3.2 QUALITY-BASED PRE-FILTTERIGG
Following OpenSora (Zheng et al. 2024), we utilize a series of filtering tools to improve data quality. including:
1. Scene Cutting. Given the extended duration of some YouTube videos (up to several hours), we employed PySceneDetecf" for scene detection and segmented the videos identified scene transitions. Each clip was constrained to length of 2 60 seconds, resulting in approximately 230K sounding video clips.
2. Aesthetic Filtering. We filter out videos with aesthetic scores (Schuhmann et al. 2021 lower than 4.5, remaining in 70K clips.
3. Optical-flow Filtering. We use UniMatch (Xu et al. 2023) to estimate the motion quality of the given videos, and remove the (static) videos whose score is lower than 0.I. This produces 46K candidates.
24

---

Published as a conference paper at ICLR 2026
Table A3: Clarification of the category taxonomy of our JavisBench.
Aspect
Category
Description and Examples
Natural Scenario
Scenes dominated by natural environments with minimal human interference.
such forests, oceans, and mountains.
"Uiban Scenario
Ouidoar spaces shaped by humia activity, including cities, villages, streets, and
parks.
Living Scenario
Indoor environments where daily human activities occur, like houses. schools."
Event Scenario
and shopping malls.
Industriil Scenatio
Work oriented spaces related industrial energy activities, such factories,
construction sites, and mines.
Virtual Scenario
Imuginative or ubstract settings, including virtual worlds, sci-fi cities, and artistic
installations.
Camera Shooting
Filmed with handheld, fixed, or drone cameras, including slow motion footage.
2D-Animate
Styles like hund-drawn animation, flat animation, cartoon styles, watercolor illustrations.
Visual Style
"3D-Aniiate
Photorealistic styles, sci- sci-i/aaagiaal effecis, CG (Computer Graphics), or steam-
punk acsthetics.
Ambient Sounds
Sounds that occur naturally in the environment, including both natural and human-influenced surroundings. This category includes sounds like wind, ruin, water flow, animal sounds, human activity (egg, traffic, construction), and urban noise.
Hiological Sounds
Sounds produced hy living creatures (e.ga ganimals, This includes vocal." izations such as barking. chirping, growling, well as non-vocal human sounds like heartbeat, and other physical noises.
Mechanical Sounds
Sounds generated by man-made machines, devices, or mechanical processes. This includes the noise of engines, motors, appliances, and any mechanical electronic noise. This category also includes malfunction sounds (egg. malfune- tioning machinery or alarms).
Sound Type
"usicall Sounds
'Sunnd' related music musical performance, including both human generated and instrument-generstee sounds and melodies. This category covers singing. instrumental performances, as well us background music used in vari- ous media formats.
Speech Sounds
Sounds generated from human speech, whether conversation, dialogue, public" speeches, debates, interviews, or monologues. This category specifically covers linguistic communication various eontexts, whether formal, informal, contentious.
Single Subject Multiplé Subject
There only primary object source producing sound in the scene.
There are multiple priniary objects that (or potentially can) make sounds in tie scene.
Spatial Composition
"Off screen Sound
The source of the sound is not visible in the scene but logically exists (e.g. a car engine outside the camera view).
Single Event
The audio contains only one event, with no overlapping sounds. For example,
"a single dog barking without background noise."
Sequential Events
There are multiple events occurring sequentially, with no overlup. For example."
Temporal Composition
"the applause begins after the music performance ends."
Simultaneous Events
Multiple audio sources are present simultaneously. such "a person speaking while music plays in the background""
4. OCR Fillering. We use DBNet (Liao et al. 2022 to detect and filter out the videos containing more than 5 text regions, resulting in a smaller 30K scope.
5. Speech Filtering. As the attemet-avvilable videos contain too much human speech (including talking and voiceover), we use FunASII to detect and remove the speech videos. In this step, there are still around 20% speech videos remaining, due the non-perfect speech detection by FunASR. We keep this part of videos in our final benchmark to ensure the diversity of video-audio sources. This step leads 22.3K sounding video clips.
D.3.3 ANNOTATION
Since most of the collected data lacks captions, we designed generic pipeline to generate captions for the video-audio pairs, and then categorize them into corresponding classes from different aspects, as illustrated in Sec. D.2
25

---

Published as a conference paper at ICLR 2026
Table A4: Data source composition before and after filtering strategies of our JavisBench
YouTube
FAVDBench
AVSBench
Landscape
AIST+
Total
Before Filtering 30,107
1,000 833
804
100
20
32,031
After Filtering 8,507
680
100
20
10,140
First, we use Qwen2-VL-72B (Wang et al. 2024c) to generate detailed captions for videos. Metadata from the data source (e.g., object labels or YouTube keywords) included in the context to enhance caption quality.
Next, we use Qwen2-Audio incorporate the previously generated video captions contextual input, it produces even more hallucinations with the input bias (Lee., wrongly identifying sound that will happen in corresponding visual scenarios but actually does not exist in the current audio).
(Chu et al. 2024 to caption the audio in detail. We do not
Then, we employ Qwen2.5-72B 1---5-2BB-nnntuue Instruct (Yang et al. 2024a) to merge video and audio captions into unified text prompt, serving as the textual condition for the JAVG task. During this process, Qwen2.5-72B is prompted to reason on the video-audio captions and identify apparent logical mistakes (e.g., there is dog in the video but Qwen2-Audio gives sounds from a car engine) or missing captions (e.g., unknown sound source).
Finally, we query Qwen2.5-722 2- again classify the data points based on the video, au- dio, and generated captions, assigning each entry to the appropriate category in the classification system. We do not prompt multimodal LLMs to do this categorization because the generative models to evaluate will only receive the text caption inputs.
After removing the logical conflict captions and classification results that fail to parse, we obtain 19.4K sounding video clips with detailed captions and hierarchical categorization results.
D.3.4 CONTENT-BASED POST-FILTERING
To build diversified and balanced benchmark to evaluate joint audio-video generation, we conduct another post-filtering based on the categorization results obtained above. In particular, we further remove videos that only contain background music and speech voice. By doing so, nearly half of the videos are filtered out, since a large part of YouTube videos rely on music and voice acting to attract viewers. After human checking, sources and fine-ge grainer category annotations, setting a new standard to facilitate a comprehensive evaluation for future joint audio-video generation (JAVG) methods. The category statistics can be found in Fig. and Tab. A4 displays the data sources before and after our filtering strategies.
obtained 10,140 samples in JavisBench with diverse data
D.4 EXTENDED SPECIFICATION ON JAVISSCORE EVALUATION SUITE
D.4.1 IMPLEMENTATION DETAILS
Given a generated video-audio pair (V, A), we use ImageBind (Girdhar et al. 2023) estimate audio-visual synchrony with following steps:
1. Temporal Segmentation via Sliding Windows. (V, .A) is chunked into several segments with 2-seconds window size and I. 1.5-seconds overlap (C = {(V,, A1), (V2, A2), (Vw, Aw) }) The 2-second window ensures compatibility with the audio encoder's minimum processing length (Girdhar et al., 2023), preventing suboptimal feature extraction. The 1.5-second overlap enhances continuity and robustness, allowing each frame be evaluated within multiple temporal contexts. This mitigates artifacts caused by abrupt segmentation boundaries.
2. Frame-wise Audio- Visual Similarity Computation. Inspired by Mao et al. (2024). we calculate the cosine similarity between each frames (F (V..1, Vi ... Vi,.= 1) and the whole audio clip A, by using ImageBind's vision and audio encoders (E,, E.). Frame-wise similarity captures fine-grained temporal dynamics, ensuring transient events (e.g. lip movements, object interactions) are accurately modeled.
3. Segment-wise Synchronization Estimation. Instead of averaging all similarity scores, select the 40% least synchronized frames (with lower similarities) and compute their mean to obtain the synchronization score for each window. By focusing the least synchronized
26

---

Published as a conference paper at ICLR 2026
frames, the metric becomes more sensitive to local resynchronization, simple mean can be biased by considerable proportion (e.g., 70%) of synchronized frames.
4. Global Synchronization Estimation. The final JavisScore is computed by averaging the window-level scores across all segments, balancing local variations while maintaining sensitivity to desynchronization patterns. Due window overlap, each video frame is evaluated multiple times, reducing the influence of outliers and providing a more stable final score.
These steps can be formulated as:
1 Wa Σ oV. A.), OV., A,) k Σ top-k (V.,), Ea (A.)))
1
Savis
(A3)
This approach effectively differentiates synchronized and desynchronized video-audio pairs.
D.4.2 VERIFICATION ON METRIC QUALITY
This paper provides quantitative comparison between the accuracy video-audio synchrony metrics. We constructed a validation dataset consisting of 3,000 audio-video pairs to evaluate the effectiveness of our proposed metric, JavisScore. The validation set is initialized by 1,000 positive (synchronous) video-audio pairs from our JavisBench dataset, and we construct around 2,000 negative pairs from three different sources:
1.
The first 1, 1,000 negative pairs are generated through online augmenaaiion/raasformaaaioo on the positive pairs. Given a synchronized video-audio pair (V+,A+), A+), we reuse the augmentation strategies in Sec. C.2.4 to generate asynchronous video-audio pairs, i.e., (V+ - or (V=,A+). The four augmentation types (video-spatial, video-temporal, audio-spatial, audio-temporal) produce 250 negative samples, resulting in the first 1,000 negative pairs.
2. The second 500 negative pairs come from separate generations. In particular, we randomly split the 1,000 positive samples into two parts, and select the first 500 samples prompt Audiol.DM2 (Liu et al.] (2024b) to generate audios conditioned solely on text captions, without accessing the corresponding videos. This naturally leads 500 asynchronous video-audio pairs.
3. The third 500 negative video-audio pairs are generated by a preliminary JAVG model condi- tioned on the other 500 text prompts from positive samples. These generated pairs were then manually annotated to determine whether they were synchronized or not. We finally obtained 411 hard negative samples with 89 positive video-audio pairs.
In the evaluation dataset, each positive sample is related constructed by online augmentation on the positive entry (the first negative source), and the other is generated from text captions of the positive data point (the second or third negative source).
negative video-audio pairs. One is
Then, we compare proposed JavisScore with the previously-used AV-Align Yaaiiv et al., 2024) metric on the 3,000 evaluation samples, and the result is displayed in Tab. A5 We calcu- late the AV-Align scores and JavisScore all the 3000 samples, and compute the AUROC value for the binary classification (ideally, a VA- synchrony metric should produce higher scores on positive video-audio pairs), which shows JavisScore achieves approximately 0.13 higher AUROC than AV-Align. Then, as each positive sample is related to 2 negative video-audio pairs,
Table A5: Comparison of VA-synchrony metrics. means we relax the criteria to Acc = ll(s 2 s
Metric
AUROC
Accuracy
Sondom-Guest
0.5000
0.5000
AV-Align
0.5296
0.5254
DeSync
0.5742
0,3961
DeSyne
0.5742
0.7797
JavisScore
0.6533
0.7514
we can calculate the prediction accuracy of 1000 x 2000 paired positive-negative samples. viewed correct when metric assigns a higher score to the positive sample than to the correspond- ing negative video-audio pair. According to Tab, [AS] our metric significantly surpasses AV-Align by 23% accuracy, demonstrating the efficacy of video-audio synchrony measurement.
Notably, AV-Align nearly performs as a random guess (with a near 0.5 AUROC and Accuracy). This is because AV-Align simply uses optical flow to capture the video dynamics, and match with audio dynamics estimated by onset detection results. there a concurrent "pulse" at the video and audio timelines, AV-Align will produce high synchrony score. However, in worl-world scenarios, the video
27

---

Published as a conference paper at ICLR 2026
Figure A6: Complex Scenario with multi-source sounds time.
Parameter Sensitivity Heatmap of AUROC
Parameter Sensitivity Heatmap of ACC
Window Parameter Conhination
Window Parameler Combination
w 5-20 w 1-2.0 w 5-20
w wro1..
75.0
65.2 65.3 65.3
65.3
65.3 65.3 65.3 65.3 63.2 65.2
w
74.3 75.1 75.1 75.1
75.0
75.0 75.1 75.1 75.0 74.9
0=11.
5-20 w---2.0
W.ord. w.
65.2
68.2
65.3
65.3 65.3
65.2 65.3 65.0 65.2
74.9 15.0 75.0
74.7 14.7 14.9 75.0 75.0
74..5
73.0
w w=000
-7..0
65.1 64.9 65.2
63.2
66.22
65. 0.6.2.66 65.2 65.2 65.1 65.2
74.5
79.8 74.4 74.4 74.5 74.5 14.5 74.6 74.6 74.6
Aveage
Topk: min--4444 40%
Topl-min-49 19%
Topl-mmin-- 50%
Topt-nath-lee min--i%%
Topk Topk-min-452
Topk-min--n 70%
Topd- Topl- 75%
Topl- 80%
Top--min--8%
Avenge
Topk Topk-min-khe 3i8--mi--0P 40%%
Topk min--1%
Topd: min 500
Topl: Topk-mins-o 40%
Topk. min- 45%%
Toppt radn- 70%
Topl Topl-minn-77 75%
Topl min--0PP
Topl-min-8555 93%%
71.0
Strategy
Strateg go.
Figure A7: Parameter sensitivity evaluation of our JavisScore. Our method presents stable and robust video-audin synchrony estimate at various settings. We finally choose (2-second window size, 1.5-second overlap, topmin-40% due to the relatively better performance.
optical flow and audio onset detection cannot precisely capture the start and end of the risual-audin event. As exemplified in Fig. A6 when person in the video speaks, the movements of their mouth are subtle and often too minimal to be effectively captured by optical flow. Moreover, in environments with strong background noise (e.g., TV programs), the onset of human talking also becomes difficult to detect. The scores generated by AV-Align are unreliable in such complex scenarios. In contrast, our proposed JavisScore leverages the high-dimensional semantic space of ImageBind (Girdhar et al. 2023) to compute synchronization at the second-level, robustly distinguishing between synchronized and unsynchronized cases. It is worth noting, however, that our metric does not achieve 100% accuracy. Developing a more precise evaluation metrie remains significant challenge in the JAVG domain, and we hope this work will inspire further advancements in this critical area.
In addition, SynchFormer (lashin al. 2024 employs a 21 -categoy classifier to estimate the temporal shift degree between video and audio, which is then used to compute synchrony score (lower is better). However, shown in Tab. A5 its discrete predictions result in poor separation between positive and negative samples, yielding low accuracy and AUROC. Only when we relax the evaluation metric Ace = I[( 2 } criterion does its accuracy approach that of JavisScore. Besides, SynchFormer is inherently limited to predicting temporal shifts and is not applicable to the diverse spatiotemporal misalignment scenarios considered in the real world (see Sec. C.2.4). Nevertheless, its approach suggests a promising direction: learning neural network-based synchrony metric via supervision, which we leave as future work.
Besides, we also conducted an ablation study to evaluate the parameter sensitivity of our JavisScore. including the sliding window size, overlap length, and score computation strategy. As suggested by sliding window size of 2, an overlap length of 1.5. and selecting the top 40% minimum strategy. However, other settings do not significantly reduce the performance of our metric. The worst accuracy, for example, still achieves around 74% and outperforms AV-Align of 52.5% by a large margin. The results further demonstrate the robustness of JavisScore metric.
Fig. A7 the optimal parameters were identified
28

---

Published as a conference paper at ICLR 2026
Event Scaveto
Vicaa Style
Scand Type
Sguria Einguatin
Tempral Compottio
tope
Souzill Limmouttoo
Tempural Compuditon
Spuial Compoiiios
Coppaiika
FAD
TV/0
Visal
Spsial Conpoiiiu
Casporikk
1A-00
Th-B
Samalo
Vicaa
Sysial Corpution
Compoliko
Dvvin Somaaoo
Virzal
Seard
Spatal Clepputtion
Tempural Corpodtion
20
Event Scmani
Visaal Stdle
Scund Type
Spuzal Chenoutiaa
Temoural Corpatiko
00/10
Lonk
We-tom
Sarff
Escal Scanaio
Voual Stye
Sound Type
Spxial Coupoiiia
Teupurl Cospaiiko
Esent Somato
Voral Stdee
Scand Type
Spasal Comrosiiio
Tempura Cumporiio
Came
Gett
Figure A8: Metric distribution on all JavisBench's 's taxonomy.
D.5 IN-DEPTH ANALYSIS OF EVALUATION RESULTS
Fig. A8 provides a more fine-grained, multi-dimensional, and comprehensive evaluation our model on JavisBench, offering in-depth analysis of the current SOTA models' limitations in real-world joint audio-video generation tasks. Based on these results, we summarize the following key insights:
Insufficient unimodal modeling capability in rare scenarios: (1) The FVD in Event Scenario (Ist row, 1st column) indicates poor video quality in industrial and virtual scenes, suggesting weaker modeling capability in these domains. (2) The FVD in Visual Style (Ist row, 2nd column) shows significant disparity between generated and real videos 3D-animate scenes, potentially due insufficient training data. (3) The FAD in Sound Type (r3c3) suggests weaker audio
29

---

Published as a conference paper at ICLR 2026
Table A6: Detailed evaluation results of ablation on model design (Tab.
AV-Quality
Text-Conee Consistency
AV-Consistency
Sv-Synchron
STDIT HiST-Sypo BICA
FVDL
KVDJ
FAD
TV:IB
TA IB1
CLIPT
CLAP
AV-IBt
CAVP1 AVlSScoret
JavisScore1
x
x
442.0
3.9
10.5
0.268
0.139
0.287
0.344
0.197
0.796
0.147
0.118
0.130
249.7
2.9
7.4
0.275
0.147
0.305
0.375
0.207
0.800
0.184
0.150
269.1
3.1
0.273
0.144
0.300
0.377
0.200
0.795
0.162
0.135
2418
2.9
7,3
0.277
0.146
0.308
0.382
0.209
0.801
0,186
0.153
Table A7: Detailed evaluation results of ablation ST-Prior design (Tab.
AV-Quality
Text-Consisteccy
AV-Consistency
AV Synchhrnn
N.
N₂
Injection
FVDJ
KVDJ
FAD1
TV-IBt
TA-IB
CLIP
CLAPP
AV-IB1
CAVP1
AVHScoref
JavisScoref
0
269.1
7.9
0.273
0.144
0.300
0.377
0,200
0.795
0.162
0,133
CresAttn
289.9
3.2
0.269
0.140
0.293
0.375
0.204
0.797
0.175
0.137
16
16
CrossArta
257.2
3.0
7.5
0,275
0.144
0.305
0.381
0.209
0.801
0.185
0.151
CrossAria
0.186
32
32
Addition
252.7
3.0
7.4
0.274
0.142
0.303
0,380
0,206
0.798
0.180
0,144
32
Modulate
255.0
0.277
0.143
0.309
0.382
0,207
0.800
0.180
0.145
modeling in ambient, biological, and mechanical categories, likely because these categories are broad. This observation aligns with Event Scenario (r3c1), where natural and industrial scenes also exhibit high FAD values.
Unimodal quality does not directly correlate with text consistency: (1) The TV-IB in Visual Style (r4c2) reveals that 2a3DDaanmmate scenes exhibit poor text-following capability, despite their unimodal quality. (2) The CLAPScore in Sound Type (r7c3) suggests weak audio-text alignment in ambient and mechanical scenes, potentially due to a lack of corresponding audio- text pairs in the first-stage (Audio Branch) training data.
Poor AV sysyncoronizttion in challenging scenarios: (1) The JavisScore in Event Scenario, Visual Style, and Sound Type (r10c1-3) show that scenarios with poor unimodal quality also suffer from weak audio-video synchronization, particularly in virtual environments, 20/31 animate styles, and musical sound types. (2) The JavisScore in Spatial and Temporal Composition (r10c4-5) indicate significantly lower AV synchronization performance for complex events (e.g., requential/simultanedd events, multiple sound sources, or off-screen sounds) compared to simpler scenes (e.g., single events with a single sound source).
In conclusion, current SOTA models still struggle with rare and complex scenarios, both in terms of video generation quality and synchronization performance. The JAVG community still faces significant challenges in bridging the gap between research models and real-world applications.
E ADDITIONAL EXPERIMENTS
FULL EVALUATION RESULTS FOR ABLATION STUDIES
In the manuscript, we several proxy metrics (Savo: Swcc SAvs) simplify the presentation when evaluating the quality, consistency, and synchrony of generated audio-video outputs. In this section, we provide the full results corresponding Tab. 3 (model-design ablations) and Tab, 4 (ST-Prior ablations) in Tab, Anland and Tab. A7 respectively, to further enhance the clarity and completeness of the paper.
E.2 EVALUATION ON AUDIO GENERATION QUALITY
Tab. A8 presents the evaluation of our model's performance in the first training stage, concerning both unimodal text-to-audio generation quality and text-following capability. Specifically, employ two datasets for evaluation:
AudioCaps test set (Kim et al. 2019) Filtered by AudioLDM2 (Liu al. 2024b). this dataset
consists of 964 samples, each 8 10 seconds long. Models are required to generate 10-second and 4-second audio clips.
30

---

Published as a conference paper at ICLR 2026
Table A8: Evaluation audio generation. After sufficient training iterations at stage 1, Jiv presents moderate audio generation performance on in-domain test set AudioCaps (Kim) al. 2019) and comparable quality on out-of-domain test set JavisBench-mini.
AudioCaps (InD, 10s)
AudioCaps (InD, 4s)
JavisBench-mini (OoD, 48)
Method
FAD1
TA-IB1
CLAPt
FADL
TA-IB
CLAPt
FAD4
TA-IB1
CLAP1
DM2
2.01
0.205
0.487
5.57
0.147
0.326
8.81
0.153
0.360
JavisDiT-audio-pp13
5.88
0.145
0.319
8.33
0.150
0.368
JavisDiT-audde -
5.19
0.164
0.356
6,23
0.141
0.301
8.11
0.152
0.381
JavisBench-mini: A subset containing 1,000 samples from JavisBench ranging from 2 to 10 seconds in length, which is also utilized in ablation studies in Sec. 5.3). Models are required to generate 4-second audio clips.
On these datasets, we first analyze the performance progression of our model across different training stages and then compare it against the current sfate-of-the-art (SOTA) model, AudioLDM22 (Liu et al. 2024b).
As shown Tab. A8 with increased training iterations (from epoch 13 to epoch 55), our model demonstrates improvements in both audio generation quality (e.g.. FAD decreases from 5.88 to 5.19 on AudioCaps) and text consistency (e.g.. TA-IB increases from 0.145 to 0.164, while CLAPScore improves from 0.368 to 0.381 on JavisBench-minij Furthermore, model achieves comparable audio generation performance to AudioL.DM2 on JavisBench-mini, further validating the effectiveness of our DiT- architecture.
Notably, our model performs worse than AudioL on AudioCaps, which can be attributed to two primary factors: (1) Training strategy: AudioLDM2 is explicitly trained for 10-second audio generation, whereas our model supports variable-length audio generation. In particular, although one can forcibly alter AMd's generation length during inference by directly modifying the shape of the noisy latent (e.g., changing it to 4 seconds), the model's 's denoising process becomes misaligned with its training distribution, leading to a significant drop generation quality, shown in Tab, [A8] In contrast, since our audio pretraining is performed under variable-length settings, the generation quality at seconds remains much closer to that of AudioL.DM2. We believe that collecting more data and training for additional iterations will further improve generation performance across different lengths. (2) Potential overfitting problem: The training data of AudioLDM2 includes the training set of AudioCaps, leading to a smaller in-domain test gap, which increases the risk of overfitting. In contrast, JavisBench is collected from more diverse real-world YouTube audio sources, and its Nomain-speciffe soundscapes Thus, performance on the out-of-domain JavisBench-mini provides a more realistic reflection of a model's 's usability in real-world scenarios.
not exposed to either AudioLDM2 or model during training.
In future iterations, we will continue enhancing the unimodal audio generation quality, as this is a crucial prerequisite for achieving precise video-audio synchronization.
E.3
INVESTIGATION OF ST-PRIOR'S 'S EXTRACTION AND REPRESENTATION
In Fig. A9 we investigate the representation capability of ST-priors from three aspects.
Firstly, since the latent representations of video (v) and audio (a) contain different numbers of spatial and temporal tokens (e.g., for 240P, 24fps, 16kHz, 4-second sounding video, E IR 100x while a E R16x61XCa). ) we also experimented with various token allocation strategies for the spatial and temporal priors. These included ratios such as 1:2 (n16x32), 1:1 (n32x32), and 2:1 (n32x16). As shown Fig. A9 the 1:1 ratio achieved the best performance. This could be attributed to the inherent disparity in the latent shapes: video latent contains significantly more spatial tokens than temporal tokens, while audio latent stays in the opposite situation. Thus, balanced 1:1 prior token ratio helps minimize bias.
Next, using the 1:1 ratio, we explored different configurations of prior numbers and dimensions. The results in Fig. A9 indicate that increasing the prior number and dimension consistently improve
31

---

Published as a conference paper at ICLR 2026
ratio
number
dimension
JavisScore
6.1599
Jove
0.1590
JoresScore
0.1590
6.1535
9.525
0,125
6.1909
0.190
0.190
6.4455
0.2475
0.2475
n16x32
n32x32
n32x16
n16x16
n32x32
n64x664
d64
d128
d256
Figure A9: Further ablation on ST-Prior
including (1) spatial-temporal token ratio, (2) token number, and (3) embedding dimension. Our default setting of "n32x32-d128" is good trade-off between performance and training cost.
Table A9: Ablation of ST-Prior loss functions. All training objectives jointly contribute to video- audio synchronization.
AV-Consistency
AV-Synchrony
Likken
Lassc
Load
Freg
IB-AVt
CAVP1
AVHScoret
AV-Alignt
JavisScore
V
X
X
X
0.190
0.799
0.167
0.097
0.133
VVA
V
X
X
0.193
0.799
0.170
0.102
0.136
V
X
0.196
0,800
0.174
0.096
0,140
X
V
0.202
0.800
0.179
0.107
0.149
V
V
V
0.209
0.801
0.186
0.122
0.153
performance, demonstrating the scalability of our approach. In this work, we chose the "n32x32+d1 28
configuration as it offers a good oafe-offf between performance and training cost, considering the diminishing returns of further scaling.
E.4 TRAINING LOSSES FOR ST-PRIOR'S ' ESTIMATION
In Sec. C.2.3
we introduce four loss functions to train our ST-Prior Estimator: (1) Token-level hinge loss: Ltokens (2) Auxiliary discriminative loss: Ldise. (3) VA-embedding discrepancy loss: Load, and (4) 1.2rrgularization loss: Creg. This section presents a detailed ablation study on the efficacy of utilized loss functions.
According to Tab. A9 Ldise brings slight improvement addition to the original Ltoken (e.g., 0.193 vs. 0.190 for IB-AV), they share the same optimization purpose- pushing the text prior anchor to the positive video-audio samples while pushing away from negative samples and differ only in the specific gradient back-propagation mechanism.
On the other hand, Lond considerably enhances the synchrony of generated video-audio pairs (e.g., 0.140 0.136 on JavisScore), thanks its ability to maximize the divergence between positive and negative video-audio samples themselves. However, Creg offers even greater benefits due to its smoother regularization, facilitating the convergence of the text prior to the positive embeddings.
Moreover, the combination of all loss functions achieves the best performance (e.g., 0.153 of JavisS- core), as they collaboratively address the same goal from different perspectives: embedding video- audio synchrony into the text prior and ensuring it captures the semantic meaning of synchronization. This comprehensively validates both our motivation and the effectiveness of our methodology.
E.5 ADDITIONAL VISCALIZATION OF ST-PRIOR INJECTION
In Fig. A10, we include more detailed generation example to illustrate how attention shifts during sequential ST-Prior injection:
32

---

Published as a conference paper at ICLR 2026
Generated Video
Generated Video
Generated Video
Spatial Attention
Spatial Attention
Spatial Attention
Generated Audio
Temporal Attention
Figure A10: Attention shifts over sequential audiovisual events during ST--rior injection.
First, when the orange cat on the left meows, the visual spatial attention concentrates on its head, which corresponds to the region highlighted by the white dashed box in the audio-attention map. Then, when the tabby cat on the right meows (indicated by the red dashed box in the audio spectrogram), the visual attention shifts the tabby's 's head, providing more direct spatial prior. Finally, when the orange cat meows again, both spatial and temporal attention shift once more, injecting updated prior information into the audio and video branches.
This example more clearly demonstrates how spatiotemporal priors are injected and shifted over sequential events, further validating the effectiveness of our method.
E.6 MORE GENERATION EXAMPLES
Fig. All presents realistic audio-visual pairs generated by our JavisDiT across diverse scenarios, including industrial, outdoor, and natural environments. These results encompass wide range of visual styles, such as camera-captured footage, 2D/3D animations, as well as various audio types, including mechanical, musical, ambient, and biological sounds. Our model effectively maintains video-audio synchronization across cases involving single or multiple sounding subjects, as well as single, sequential, or simultaneous sounding events, All multimedia resources are available in the supplementary materials.
E.7 DISCUSSION; EXTENSION TO X-CONDITIONAL GENERATION
Built on diffusion models with transformers (DiT), our JavisDiT can be easily extended to support various conditional video-audio generation tasks, shown in Fig, A12 Inspired by UL2 (Tay et al. 2022) and OpenSora (Zheng et al. 2024), we propose dynamic masking strategy to support video and audio conditioning, from the basic (I) rext-o--audiovideer (t2av) generation (II) audio-to-video (a2v) generation, (III) video-t--audio (v2a) generation, (IV) audio-inage-to-videe (ai2v) generation, (V) image-to-audoo-vilee (i2av) generation, and (VT)
videoo-xtension (av_ext) generation. Note that the text condition still works in all kinds of conditional generations, which provides both coarse- coarse-grained global semantic embeddings and fine- sine-grained spatio-tempora priors on targeted visual-sounding events. In particular, unmask the specific video and frames to be conditioned on for X-conditional generation. During both model training and inference, unmasked frames will have timestep 0, while others remain the same (t):
Unmasking all video/audio frames for v2a/a2v generation. Unmasking the first frame of the video (and all audio frames) for i2av (and ia2v) generation.
33

---

Published as a conference paper at ICLR 2026
Industral Camem Ambieni, Mochanical
Muliiple Simultaneous
In fictery, person welding mettal while wearing protective gear. Various tools and equipenent are visible in background. The sound weldingg promineni, accompaneed continnous hum of heavy ongine and occasional generic impact sounds.
Urban Camera SpechhSp
Multiple Simultancous
lange, enthusinsiic illuminated vibrunt lights. live performance features steel gultar melody. scoustic rhythm and passionate mule vocals. Crowd wotses add to the lively ambiance. making both emotional and spirited.
Natural Multiple Simulcancous
Ambiien,Muiee Musical
An scvial shoe captures the rugged roastal landicope with drawaric cliff rocky outcrops. the meks. A tumbourie adds rhythmic the beal. The combication the natural apheat music entances the ovenll serene the scene.
against
countal sounds
Urban 3DAnnima Ambiccs,Mechmiiee
Multiple Simulzancous
ohaori bunlefield with tokk the
background fires, causing lorge explosion and fire one of the realistic 3D
buildings The setting aulmation. Soldiers, wearing and militrry unifoons, are visible, with aiming weapon the foroground.
Virtual DD-Animtte Ambieen, Musical
Multipio, Off sereen Simmlhnncou
fantustical lundvcupe features colorful rock formations, flowing river, and autamnal trees set against a surreal backdrop
large, glowing planet the sky. The scene is unimated vivid 20, blending natural and alien elements, music playing background
pos
Natural Camern Biological
Single Single
with light-oolored far, possibly white or cream, playfullyy on the ground surrounded by fallen leaves and nelgs, It round, expressive facce large, dark eyes and small nose, and its mouth open if smilling making sound.
young
Natural Camera Ambient AmbientBinlogical
Multiple Simultancous
small possibly urvs, perches on a branch in a dense forest, preening
feathers with brownish plumage and short Ruia falls and wind blows through the trees, creating soothing backdrop the bird cleuns and arranges feathers.
Figure A11: Extensive JAVG cases diverse event scenarios, visual styles, audio types, sounding subjects, and temporal compositions. Our JavisDiT achieves high-quality and text-consistency for single-modality generation and keeps a good video-audio synchronization.
Unmasking a preceding part of video and audio frames for va-extension.
We view the integration of all audio-video interactive generation tasks in one unified model as our future work.
34

---

Published as a conference paper at ICLR 2026
10 c2s
a2y
v2l
videoo
[i]) av coi
mudoo
audio
mudio
Patches in one video frame without nolse (timeReppr)) ())
Patches one vide fiamee with moise (limoeupet)
Patches in one audoo frunc withou motec (limedep-1))
audio meiiss (tincelopet)
Figure A12: Masking strategies for X-conditional generation. The DiT architecture allows feasible conditional video-audio generation by replacing the noisy latent representation with reference videos and/or audios with specific strategies during training and inference.
35


Reviews:
[{"review_id": "aHqUYPOaeo", "reviewer": "Reviewer_nc3c", "summary": "JavisDiT is an end-to-end Diffusion Transformer ($\\text{DiT}$) architecture that simultaneously generates high-quality video and audio content. The core innovation is the Hierarchical Spatial-Temporal Synchronized Prior ($\\text{HiST-Sypo}$) Estimator. This module extracts both coarse-grained semantic priors (overall framework) and fine-grained spatio-temporal priors (timing and location of events) from the input text to guide the synchronization of the generated video and audio.\n\nKey Contributions:\n\nNovel JAVG Architecture (JavisDiT): A unified $\\text{DiT}$-based system that leverages spatio-temporal self-attention, cross-attention, and a Multi-Modality Bidirectional Cross-Attention ($\\text{MM-BiCrossAttn}$) module to achieve high-quality joint generation.\n\nHierarchical Synchronization Mechanism ($\\text{HiST-Sypo}$): A fine-grained alignment module that estimates and injects text-derived spatial priors (where an event occurs) and temporal priors (when an event starts/stops) into the generation process.\n\nNew Benchmark ($\\text{JavisBench}$) and Metric ($\\text{JavisScore}$): The paper introduces $\\text{JavisBench}$ ($\\sim$10,140 samples), a challenging dataset focused on complex, multi-event, real-world scenarios (including sequential and simultaneous events). It also proposes $\\text{JavisScore}$, a temporal-aware semantic metric designed to more robustly evaluate fine-grained synchronization than previous methods.", "questions": "Robustness to New Language (Zero-Shot/Few-Shot): The HiST-Sypo relies on an ImageBind encoder trained on text-image pairs. How robust is the HiST-Sypo estimator to novel descriptive phrases (e.g., neologisms, highly contextual language) that were not present in its training corpus? Does the learned spatio-temporal prior generalize effectively when the text is complex and abstract?\n\nNecessity of Two-Stage Prior Training: The ST-Prior Estimator is trained separately (Stage 2) using a complex contrastive learning pipeline (four loss functions, negative sampling strategies). Was a simpler end-to-end training strategy (e.g., training the estimator jointly with the full JAVG model using only the JAVG loss and minimizing a simpler feature distance) attempted? If so, why did it perform worse than this multi-stage approach?\n\nVisualization of Fine-Grained Alignment: Figure 7 is a compelling visualization of the Spatial and Temporal Attention maps, confirming the model focuses on the correct object (\"bubbles\" vs. \"diver\"). Could the authors provide a similar visualization for a multi-event, sequential or simultaneous case (e.g., like the one in Figure 1, robot and dog tussle, then aliens talk) to demonstrate how the attention maps shift over time to capture the different spatial and temporal priors injected?\n\nJavisScore Parameter Selection Justification: The authors selected Topk-min-40% as the synchronization estimation strategy (focusing on the 40% least synchronized frames). The ablation (Fig. A6) shows only minor variations in AUROC across most Topk thresholds. Why was 40% chosen over 50% or 60%? Is there a theoretical or perceptual reason for focusing on the hardest 40% of the clip, and did human evaluation confirm that desynchronization is most noticeable at these specific thresholds?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The combination of $\\text{DiT}$ as a backbone for joint audio-video generation with the Hierarchical Spatio-Temporal Prior ($\\text{HiST-Sypo}$) Estimator is highly novel. Previous JAVG methods lacked explicit fine-grained spatio-temporal modeling. The $\\text{JavisScore}$ metric, which targets the least synchronized frames to increase sensitivity to local desynchronization, is also an original contribution to evaluation methodology.\n\nThe system design, including the detailed module structure (Figure 2) and the three-stage training strategy (pretraining, $\\text{ST-Prior}$ training with contrastive loss, $\\text{JAVG}$ training), is robust. The ablations demonstrate that the $\\text{HiST-Sypo}$ estimator provides a significant and consistent gain in $\\text{AV-Consistency}$ and $\\text{AV-Synchrony}$ compared to simple bidirectional cross-attention ($\\text{BICA}$). Furthermore, the empirical verification confirms that $\\text{JavisScore}$ is substantially more reliable ($\\sim$23% higher accuracy) than $\\text{AV-Align}$ in distinguishing synchronous from asynchronous pairs.\n\nThe paper clearly articulates the dual challenges of high-quality generation and fine-grained synchronization. The distinction between $\\text{Spatial Alignment}$ and $\\text{Temporal Alignment}$ is precisely defined. The framework is well-diagrammed, and the appendix provides ample technical detail on data curation, augmentation for negative samples, and loss functions.\n\nThis work significantly elevates the bar for $\\text{JAVG}$ research. $\\text{JavisDiT}$ achieves $\\text{SOTA}$ performance across both established and the new, challenging $\\text{JavisBench}$ dataset, setting a new standard for generation quality and synchronization coherence. The introduction of $\\text{JavisBench}$, focused on multi-event complexity (75% multiple events, 57% simultaneous events), directly addresses the \"out-of-domain\" issue that hinders the practical applicability of current models.", "weaknesses": "Computational Cost and Efficiency: The paper acknowledges the high computational overhead of the $\\text{DiT}$-based architecture and diffusion process9. The latency analysis in Table $\\text{A1}$ indicates that generating a small $240\\text{P}, 4\\text{s}$ clip takes $\\mathbf{30}$ seconds on an $\\text{H}100 \\text{ GPU}$, which is prohibitive for real-time or fast creative applications. The training of the full JAVG model takes $\\mathbf{256}$ GPU days on $\\text{H}100$10. Suggestion: Discuss explicit strategies for accelerating inference (e.g., consistency models or distilled sampling methods) as a crucial next step to transition the model from research to practical use.\n\nDataset Leakage and Bias: While the authors took steps to avoid leakage by collecting videos between June and December 2024 and filtering 11, they trained the audio branch on vast public datasets ($\\sim 788\\text{K}$ entries) including $\\text{AudioSet}$ and $\\text{VGGSound}$12. These datasets are known to contain biases (e.g., $\\text{off-screen}$ sounds) and may overlap non-trivially with the test sets of cascaded models (which often use V2A models trained on these very datasets). Suggestion: Explicitly discuss how the $\\text{HiST-Sypo}$ training (Stage 2) mitigates biases learned by the audio branch during pretraining (Stage 1).\n\nAmbiguity in Prior Estimation: The fine-grained $\\text{ST-Prior}$ estimation adopts a $\\text{VAE}$-like sampling strategy to model the variability of events for the same text prompt: $(p_{s},p_{t})\\leftarrow\\mathcal{P}_{\\phi}(s;\\epsilon)$13131313. However, it's unclear how much of the final generation is controlled by the sampled noise ($\\epsilon$) vs. the core semantics. Suggestion: Discuss or ablate the influence of the sampling variance ($\\mathcal{P}_{\\phi}$'s variance output) on the final generated synchronization, specifically for multi-event or off-sc", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2edlLR9MO1", "reviewer": "Reviewer_Fny1", "summary": "1. The paper proposes a joint audio-video generation model named JavisDiT, which is based on the Diffusion Transformer (DiT) architecture and aims to simultaneously generate high-quality and synchronized audio-video content from text prompts.\n2. The paper designs a core module, the Hierarchical Spatial-Temporal Synchronized Prior Estimator, which extracts both global coarse-grained semantic and local fine-grained spatio-temporal priors from text. These priors are then injected into the DiT module to guide the precise spatial and temporal synchronization of the audio and video.\n3. Addressing the lack of scene diversity and complexity in existing benchmarks, the authors constructed a new and more challenging benchmark dataset, JavisBench. If this dataset were to be made open-source, it would be beneficial to the community's development.\n4. To address the deficiencies of existing metrics (such as AV-Align) , the paper introduces JavisScore. This new metric uses a temporal-aware semantic alignment mechanism to robustly measure synchronization in complex real-world scenarios.", "questions": "1. The HiST-Sypo estimator uses ImageBind's text encoder, while the proposed JavisScore metric also relies on ImageBind to measure audio-visual synchronization. Does this not create a 'circular' evaluation, where the model is effectively trained to optimize for the feature space of the metric that is supposed to be judging it?\n2. Regarding the audio autoencoder, why did the authors opt for a Mel-spectrogram based VAE (which necessitates a vocoder) instead of a waveform-based VAE (e.g., Wave-VAE)? The choice of a Mel-VAE introduces an inherent quality loss from the vocoder, which could be a critical bottleneck for the final audio generation quality.\n3. The paper claims to support variable-length audio generation, possibly via dynamic temporal masking. How is this practically implemented during inference? Given that the main evaluations are on 4-second videos, what evidence is there that this mechanism can robustly scale to longer durations (e.g., 10 seconds) while maintaining quality and synchronization?\n4. In the ablation studies (Sec 5.3, line 795), the paper introduces normalized scores ($S_{AVQ}$, $S_{AVC}$, $S_{AVS}$) to simplify the results. What is the justification for the specific weighting factors used in these formulas? These equations appear arbitrary and lack a clear theoretical or empirical basis, making the ablation results difficult to interpret.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper's motivation is clear and precisely targets the two core challenges in the field of Joint Audio-Video Generation (JAVG): ensuring high-quality generation of audio and video, and maintaining perfect synchronization between the two modalities. It is rare for a single work to contribute a model, dataset, and metric simultaneously. This represents a significant potential to advance the field and, as an academic work, is sufficiently compelling.", "weaknesses": "1. There is a clear disconnect between the paper's core claims and the qualitative results provided in the supplementary demo. The paper claims to solve high-quality generation and perfect synchronization, but the demos fail on both counts. The visual quality is significantly lower than the level of current video generation models, and audio-video synchronization is not well demonstrated in these demos, often appearing as coarse \"scene-ambience\" rather than the claimed \"fine-grained\" alignment. Furthermore, the number of samples is insufficient to cover the breadth of capabilities claimed. Critically, the demos lack the most essential evidence: direct comparisons against baseline methods (e.g., \"T2V + V2A\" or \"T2A + A2V\") for the same prompt. While the authors provide objective metrics in Table A8, the corresponding qualitative examples to visually substantiate these claims are missing.\n2. The paper lacks sufficient subjective (human) evaluation. This is a necessary component for measuring video quality, audio quality, and audio-video alignment, and its absence is a significant omission.\n3. The authors' justification for their model's weaker audio performance (e.g., in line 1538) is unconvincing. Attributing this weakness to supporting \"variable-length audio generation\" is a flawed argument, as the baseline AudioLDM 2 also supports this feature. Additionally, while the authors' concern about AudioLDM 2 training on AudioCaps is valid, their use of the term \"data leakage\" is imprecise.\n4. The multi-stage training strategy is overly complex. Does the complex multi-stage training strategy require significant manual intervention and tuning when applied to new datasets.\n5. The paper's readability is poor due to dense prose. For example, the term \"Fine-Grained Spatial-Temporal Self-Attention Cross-Attention\" (line 89) is structurally confusing. This confusion is amplified by the inconsistent naming in Figure 2 (e.g., \"Fine-Grained ST-CrossAttn\" and \"ST-SelfAttn\"), which hinders a clear understanding of the architecture. While the authors state in the Appendix (A.4) that LLMs were used \"solely as writing assistants,\" we believe the paper's over-reliance on such tools has resulted in these convoluted descriptions and reduced overall readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tkTAHCh0QV", "reviewer": "Reviewer_giZt", "summary": "This paper addresses joint audio-video generation, focusing on two key objectives: (1) achieving strong generative performance for each individual modality, and (2) ensuring fine-grained synchronization between audio and video. To meet these goals, the authors propose JavisDiT, a diffusion transformer (DiT)-based model capable of generating high-quality audio and video. To enhance cross-modal alignment, they further introduce a hierarchical spatial-temporal synchronized prior estimation module. In addition, the paper presents a new benchmark, JavisBench, consisting of high-quality text-annotated sounding videos for evaluating joint audio-video generation.", "questions": "1. Training such a model requires substantial computational resources. Therefore, it would be valuable to further explore the broader potential of JavisDiT, for example, by examining its applicability to other related tasks.\n\n2. The robustness of the model with respect to the quality of the training dataset should also be discussed, as dataset noise or imbalance could significantly affect performance and generalization.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The two identified challenges—developing a strong backbone for joint audio-video generation and achieving fine-grained audio-video synchronization—are well-motivated and reasonable.\n\n2. Both the proposed JavisDiT model and the accompanying benchmark are valuable contributions that can provide meaningful insights to the research community.\n\n3. The experimental evaluation is comprehensive and well-conducted.", "weaknesses": "1. Although the JavisDiT model is clearly described, its architectural design is not entirely intuitive. For example, the rationale behind the specific designs of the spatial-temporal self-attention and spatial-temporal cross-attention modules is not well explained. Providing more insights into the design choices would make the model structure easier to understand.\n\n2. The proposed JavisScore is reasonable. It might be interesting to explore whether it could be extended in a hierarchical manner—for example, by incorporating multiple window sizes for evaluation.\n\n3. Given the crucial role of the training dataset in determining both model accuracy and generalization, it would be helpful to clarify the main criteria used to ensure dataset validity. Additionally, it would be valuable to discuss how JavisDiT performs when potential biases arise from the heavy reliance on VLM-generated video captions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X09aJdDWEA", "reviewer": "Reviewer_AE6y", "summary": "This paper presents JavisDiT, an end-to-end audio-video synchronized generation model based on the Diffusion Transformer architecture. By introducing a Hierarchical Spatial-Temporal Synchronized Prior (HiST-Sypo), the model achieves fine-grained alignment between visual and auditory signals, significantly improving spatial-temporal consistency and overall generation quality. The authors also construct a large-scale dataset, JavisBench, and propose a new synchronization evaluation metric, JavisScore, which enhances the systematicity and objectivity of the study. Experimental results show that JavisDiT outperforms existing methods across multiple benchmarks. Overall, this work is solid with new innovations and extensive experiments, demonstrating potential application in multimodal generation.", "questions": "- An open question is whether it would be possible to use much less data to achieve comparable performance?\n\n- Regarding the text semantic understanding module: In related works such as Tango [1] (text-to-audio model) and MultiTalk [2] (speech-based audio-visual generation model), the text encoder is often based on T5-like large language models [3,4], which can capture richer semantic representations. Has the author considered adopting similar LLM architectures to further enhance text semantic modeling capability?\n\n[1]Text-to-audio generation using instruction guided latent diffusion model. ACM MM.\n\n[2]  Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation. arXiv preprint arXiv:2505.22647.\n\n[3] Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70), pp.1-53.\n\n[4] UniMax: Fairer and More Effective Language Sampling for Large-Scale Multilingual Pretraining. ICLR.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The research topic in this paper is meaningful and valuable. Audio-video joint generation (JAVG) is a promising emerging field, and an open-sourced JAVG method is appreciated.\n- The proposed model exhibits some novel ideas. In particular, the proposed Hierarchical Spatial-Temporal Synchronized Prior (HiST-Sypo) enables fine-grained modeling of spatial-temporal synchronization relationships in generation, which is a new and promising attempt.\n- The data and evaluation contributions are also great. The proposed JavisBench benchmark and JavisScore metric may provide valuable resources and evaluation standards for future research in the community.", "weaknesses": "Given the clear motivation and solid contributions in method, data, and benchmark, I have no major concerns. I would like to share some questions or suggestions for further improvement. Specifically,\n\n- The model structure may be complex and resource-intensive. The dual-branch Diffusion Transformer architecture for audio and video involves a large number of parameters, leading to potentially high training and inference costs. The proposed HiST-Sypo requires additional pre-training. It is recommended to include a complexity and resource consumption analysis in both training and inference.\n\n- The paper lacks failure case analysis. Although the proposed method is effective, it would be better to include typical failure cases and corresponding discussions, which would help reveal the model’s limitations and potential directions for improvement.\n\n- There is a slight overclaim in some parts of the writing. The current work mainly focuses on sound effects generation and has not yet been extended to speech audio. It is suggested to avoid overextended descriptions of timbre or speech-like attributes in the Introduction section to maintain precision and academic rigor.\n\n- Minor issues: 1) The appendix contains comprehensive details. But, it may be better to move the ablation table 4 into the appendix while adding more details back to the main paper (eg, to further explain or validate the key design of HiST-Sypo).  2) Typo: '240P4s' in the Caption of Table A1 at Line 810 should be '240P 4s'?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)