JavisDiT++: Unified Modeling and Optimization for Joint Audio-Video Generation
Authors: Kai Liu
Yanhao Zheng
Kai Wang
Shengqiong Wu
Rongjunchen Zhang
Jiebo Luo
Dimitrios Hatzinakos
Ziwei Liu
Hao Fei
Tat-Seng Chua
Venue: ICLR 2026 Poster (2026)
Keywords: AIGC
Diffusion Model
Sounding Video Generation
Text-to-Audio-Video Generation
Joint Audio-Video Generation
Video Generation
Primary area: generative models
TL;DR: We introduce JavisDiT++, a concise yet powerful DiT model to generate semantically and temporally aligned sounding videos with textual conditions.

Abstract: Recent AIGC advances have rapidly expanded from text-to-image generation toward high-quality multimodal synthesis across video and audio. Within this context, joint audio-video generation (JAVG) has emerged as a fundamental task that produces synchronized and semantically aligned sound and vision from textual descriptions. However, compared with advanced commercial models such as Veo3, existing open-source methods still suffer from limitations in generation quality, temporal synchrony, and alignment with human preferences. To bridge the gap, this paper presents JavisDiT++, a concise yet powerful framework for efficient and effective JAVG. First, we introduce a modality-specific mixture-of-experts (MS-MoE) design that enables cross-modal interaction efficacy while enhancing single-modal generation quality. Then, we propose a temporal-aligned RoPE (TA-RoPE) strategy to achieve explicit, frame-level synchronization between audio and video tokens. Besides, we develop an audio-video direct preference optimization (AV-DPO) method to align model outputs with human preference across quality, consistency, and synchrony dimensions. Built upon Wan2.1-1.3B-T2V, our model achieves state-of-the-art performance merely with around 1M public training entries, significantly outperforming prior approaches in both qualitative and quantitative evaluations. Comprehensive ablation studies have been conducted to validate the effectiveness of our proposed modules. All the code, model, and dataset are released at https://JavisVerse.github.io/JavisDiT2-page.

Published as a conference paper at ICLR 2026
JAVISDIT++: UNIFIED MODELING AND OPTI- MIZATION FOR JOINT AUDIO- VIDEO GENERATION
Kai Liu'; Yanhao Zheng'; Kai Wang Shengqiong Wu?, Rongjunchen Zhang', Jiebo Luo Dimitrios Hatzinakos", Ziwei Liu", Hao Fei Tat-Seng Chua"
'Zhejiang University, "National University of Singapore, Uniiersityy of Toronto, HiiThink Research, University of Rochester, Nanyang Technological University
I| II=-I[0== ||0== (Im[))
Prompt: Several pigeons are gathered a rocky shore near the water. One pigeon splashes energetically, sending ripples, while the others stand watching. The sound of splashing water, gurgling, and flapping wings fills the air.
Prompt: young girl plays the piano.
++] tiittet wiieettekkkes pop
Prompt:
brown bear is walking towards the camera, growling natural with greenery in the background.
Prompt: sports car races around track bordered by grass and fences. The engine roars through the air.
Figure 1: Realistic and diversified joint audio-video generation examples by our JavisDiT++ model.
ABSTRACT
AIGC has rapidly expanded from text-to-image generation toward high-quality multimodal synthesis across video and audio. Within this context, joint audio- video generation (JAVG) has emerged as fundamental task that produces syn- chronized and semantically aligned sound and vision from textual descriptions. However, compared with advanced commercial models such as Veo3, existing open-source methods still suffer from limitations in generation quality, temporal synchrony, and alignment with human preferences. To bridge the gap, this paper presents JavisDiT++. concise yet powerful framework for unified modeling and optimization of JAVG. First, we introduce a modality-sseecinf mixture-on of of-experts (MS-MoB) design that enables cross-modal interaction efficacy while enhancing single-modal generation quality. Then, we propose temporal-atigned RoPE (TA- RoPE) strategy achieve explicit, frame-level synchronization between audio and video tokens. Besides, we develop an audio-video direct preference optimization (AV-DPO) method align model outputs with human preference across quality,
"qqual contribution. Part of this work was done HiThink Research. Email: kail @ zju.edu.co Corresponding author. Email: haofei7419 gmail.com

---

Published as a conference paper at ICLR 2026
consistency, and synchrony dimensions. Built upon Wan2.1-1.3B-T2 our model achieves state-of-the-ar performance merely with around 1M public training en- tries, significantly outperforming prior approaches in both qualitative and quanti- tative evaluations. Comprehensive ablation studies have been conducted to vali- date the effectiveness of our proposed modules. All the code, model, and dataset github. www.wioaJJaia.... page
are released at
INTRODUCTION
Al-Generated Content (AIGC) has evolved from text-to-image generation towards more complex domains such as video and audio (Wan et al. 2025: Zheng al.
2024 Liu et al. 2024b Jiang
at 2025). The intrinsic correlation between audio and video moda ities has also been increasingl
explored, giving rise related research video-to-audia (Xing et al.
2024 Cheng al. 2025 Tian
iet al. 2025) and audio-to-video (Lyu et al. 2024] Yariv et al. 2024b Gao et al. 2025b) generation.
With the rapidly growing demands for AIGC domains like short videos, film, gaming, and VR.
vidio generation (JAVG) from textual inputs has become increasingly crucial in this era (Wang
al. 2025a
Lyu et al. 2025).
MM-Diffusion (Ruan et al., 2023) represents series of early works that study unconditional audio-video generation, with a scope limited to natural landscapes (Landscape (Lee et al. ) and human dancing (AIST+ (Li et al. 2021).. Then, Uniform (Zhao et al.) 2025) extended the task to label-name-con -ame--ondiiiooneddd JAVG and evaluated the model on the relatively
JavisDiT
V-Quality
UniVonse-1
Chars
Veo-3
A-Qunlity
AV-Sync
3.1
0.6%
broader VGGSound dataset (Chen et al. 2020).
More recently, JavisDiT (Liu et al.
2025c)
and Universe-1 (Wang et al.) (2025a) have be- gun to employ free-form text prompts as in- puts, systematically exploring sounding video generation in real-world scenarios. Despite these efforts, current approaches still struggle to produce high-quality, temporally synchro- nized sounding videos compared with advanced proprictary models such as Veo3 (DeepMind) Figure 2: Comparison with recent JAVG models. 2025). Fig. 2 quantitatively shows the gap be- tween previous methods and Veo3 (details are provided in Sec. B.3 indicating current JAVG meth- ods fail to capture human preferences regarding the aesthetics and harmony of audio-visual content.
TV-Align
AV-Align
TA-Align
To bridge the gap. provide unified perspective of JAVG modeling and optimization to generate high-quality, beller-synchronized, and suman--refferenc--llieee
sounding videos.
First, improve generation quality, introduce modality-specific mixture-of-experts (MS- MoE) module, where audio and video tokens exchange information through shared multi-head self- attention layers, and then aggregate intra-modal information via two separate FFN layers (Deng eet al. 2025) This architecture enhances single-modality generation quality compared with Uni- form (Zaoo al. 2025), which processes aggregated audio-video tokens using single FFN mod- ule. On the other hand, MS-MoE is simpler, more efficient, and more unified than dual-DiT plus visuall interaction block designs such as JavisDiT (Liu et al. (025c) and Universe-1 (Wang
et al. 2025a). Sec. C provides detailed architectural comparison with related works.
Second, to enhance audio-video synchrony, we propose an temporal-aligned multimodal RoPE (TA-RoPE) strategy, which aligns the position IDs of audio and video tokens unified temporal axis (Xu et al. 2025), enabling direct and frame-level fine-grained temporal synchronization. On one hand, this strategy modulates audio-vides temporal synchrony more explicitly and effectively against the ST-Prior in JavisDiT (Liu et al. 2025c) and the Stitching strategy in Universe-1 (Wang ee al. 2025a). On the other hand, RoPR is actually compatible with these methods it can be combined with ST-Prior and frame-level cross-attention for slightly better performance. However, ultimately abandon these combinations maintain overall simplicity and efficiency.
2

---

Published as a conference paper at ICLR 2026
Finally, capture human preference, we design audio-video direct preference optimization (AV-DPO) method to align the model with collected preference data (Liu et al.) 2025b), further en- hancing video-audio generation quality and synchronization. To curate robust preference data, leverage diverse reward models to comprehensively evaluate the generated audio and video across multiple dimensions (e.g., quality, consistency, and synchrony) and then adopt normalized modality- aware ranking to select winning-losing pairs. To the best of our knowledge, we are the first to in- troduce preference alignment into JAVG, enabling the model generate high-quality, synchronized sounding videos, more faithfully aligned with the input text and human preferences.
With the above techniques, our + model built upon Wan2.1-1.3B-1 2025), and is efficiently trained using only 780K diversified audio-text pairs (Liu et al.) 360K high-quality sounding videos (Mao et al. 2024), supporting arbitrary duration and resolution ranging from 2-5 seconds and 240p-4800 resolution across different aspect ratios. It achieves state- of-the-art performance, surpassing JavisDiT (Liu et al., (025c) and Universe-1 (Wang et al.) (025a)) across both qualitative and quantitative evaluations various dimensions. Extensive ablation stud- ies demonstrate the effectiveness and rationality of the proposed modules. We hope this work will set milestone for the field of native joint audio-video generation and provide further inspiration in the field.
(Wan et al. 2025c) and
In summary, main contributions are threefold:
We propose a concise JAVG model architecture, which employs a modality-specific MoE strat- egy for efficient and high-quality audio-video generation, and introduces temporally aligned RoPE to achieve precise temporal synchronization.
We the first to introduce human preference alignment into JAVG, with the AV-DPO algo- rithm to consistently improve quality, consistency, and synchrony of sounding video generation.
We train a state-of-the-ee JAVG model using only IM public data entries, hoping to provide a milestone with new inspirations for the field of native joint audio-vides generation.
2 RELATED WORK
Joint Audio-Videe Generation. Recent JAVG approaches have taken several forms. Some studies use a unified representation, projecting both modalities into shared latent space, such as CoDi (Tang et al. 2023; 2024). MM-LDM (Sun et al.] 2024). and UniForm (Zhao et al. 2025). How- ever, this strong constraint can cause moodllity-speciff information loss and provide insufficient temporal control. Another line of studies performs intermediate fusion. MM-Diffusion (Ruan et al.) 2023]. SyncFlow (Liu al.) [0024a), and AV-DIT (Wang et al.) (2024) exchange information between modalities within the model's layers using cross-modal attention or adapters. Other methods like Seeing (Xing et al.) (024)) and MMDisCo (Hayakawa et al. 2024) use online discriminators to adjust the outputs of separately pre- traited models. More recently, JavisDiT (Liu et al. (225c) develops two-stream DiT with a spatio-temport prior estimator to guide alignment, and Uni Verse-1 (Wang let al. 2025a) leverages two pretrained DiT's with stitching strategy for crossmodal information exchange. However, those methods (Guan et al. 2025) Hu et al. 2025) Low et al.) (025) HaCohen al. 2026) all rely on complicated (symmetrical or asymmetrical) model architectures and ad-hoc implicit synchrony modulation, hindering the scalability of unified sounding video generation.
RL in Generative Models. Reinforcement learning (RL) has been widely applied align genera- tive models with human preferences. Early approaches use policy-based algorithms such as Proxi- mal Policy Optimization (PPO) improve diffusion models (Black et al. [023)] Fan et al.) 2023
and Direct Preference Optimization (DPO) is subsequently introduced to align text-to-image gener- ation without explicit reward models (Wallace et al. 2024 Yang al. (024).. This area continues to develop, with recent work improving DPO (Dang et al. 2025] Huang et al. 2025a) and adopting newer strategies like Group-wise Ranking Preference Optimization (GRPO) WWang et al., 2025b
Xue et al. 2025 Liu et al. 2025al Yuan et al. 2025). Although these alignment techniques are now widely used for other modalities like video and audio (Zhang et al. 2023 Liu et al. 2025b d Furuta et al. 2024; Wu et al.) 2025 Gao al. 2025a| Chen et al. 2025), their application to complex, cross-modal tasks remains limited. To the best of our knowledge, we are the first to successfully
apply a preference alignment algorithm to the field of joint audio-video generation.
3

---

Published as a conference paper at ICLR 2026
22 ROPE
E
Temp Aligg
Norm
Video FFN
Cross Attention
Add & Gather
Add Scatter
Norn
Self' Attention
Add
Fa
Norn
Audio
FFN
D2
Tirestep
MLP
Video Token
Audio Token
Toke
MLP
Figure 3: Architecture of JavisDiT++. We use shared attention layers to encourage audio-visse visual mu- tual information modeling. with modallity-speciinn FFN layers to enhance intra- stral aggregation. The Temporal-Alignee RoPE strategy is applied to ensure audio-video synchrony. The audio/video embedder layer and prediction head that bridge DiT and VAEs are hidden for simplicity.
3 METHODOL.OGY
3.1 PRELIMINARY
Let po denote the target data distribution and pi be a simple prior distribution (e.g. standard Guassian (00, I)). Rectified Flow (Liu et al. 2023) constructs straight-line paths from noise samples X1 ~ P1 to data samples X₂ ~ Pos where the trajectory x, at time E [o, 1] is defined (1 x)x0 + tx1. The corresponding target velocity field becomes X1 Xo. A neural
Flow Matching,
network Ve (Xe, t) is then trained to regress this target field by minimizing the following objective:
(P) = Ee- [0.10.11,,x0 xo [xl 1) - v|
(1)
Once trained, new samples can be generated by solving the ordinary differential equation dx/dt (x(x 1) from / 0 1,, starting from initial noise sample Xo ~ po-
Joint Audio-Video Generation. The goal of JAVG is to model the conditional distribution p(A, V c). where V RT. xxxxHxxMM denotes video with T, frames of resolution H x W. and A R. T.T. denotes the audio in a mel-spectrogram with T, temporal steps and M frequency bins. Given condition e (e.g., text prompt), model po trained generate synchronized audio-video pairs:
(x" (x" = vo(x) tt,
GPM (0) E...
[||x* v011 + ||
(2)
Rotary Position Embedding (RoPE). In video generation Wan al., 2025 position IDs assigned along temporal (T), height (H), and width (W) dimensions, and their rotational position embeddings applied to queries and keys in attention layers:
R(t, h, w) (Rr Ry (h); [w)]]]
(g. (R(t,, his wi)q, R(t,, hy, wy ky))
(3)
This design captures the relative positional relationships of tokens across all three dimensions.
3.2 THE DIT MODEL ARCHITECTURE
From Sual-Stream DIT to Unified Backbone. Unlike dual-stream DiT frameworks such as JavisDiT (Liu et al. 2025c) and UniVerse- (Wang et al. 2025a), this paper aims to design concise, efficient, and unified DiT architecture to jointly process audio and video tokens, ensuring better scalability. As shown in Fig. first flatten and concatenate video and audio tokens, feeding them into subsequent modules for full self-attention to enable dense and rich cross-modal interaction. The tokens then separated and passed through modality-specifie FFNs, which ensure sufficient
4

---

Published as a conference paper at ICLR 2026
00,00
00011
00.2
00.3
11000
(10.1)
0,0
1,03
200
20.1
202
20,33
Video
01,00
01.
0
1.20
0,3
011,0
011
012
011,3
210
21,
212
21,33
02,00
0.2,1
02
0.2
1,200
(12,0
1,22
(123
220
2221
22
223
Unified
Timeline
2
(0:3.4)
04444
(1,5,4)
(1,6,)
(27/4)
228
Audio
(0.35)
04.5)
(1,5,5)
(1.6.)
(27,5)
(2,8.))
Figure 4: Illustration of temporal-aligned rotary position encoding for video and audio tokens.
intra-modal aggregation. The video VAE from Wan2.1 (Wan al. 2025) and audio VAE from AudioLDM2 (Liu et al. 2024b) are retained and frozen during the whole training process.
SSpelity Feed-Forward Network. In contrast conventional MoE architectures with dynamic routing (Cai et al. 2025), our MS-FFN (or MS-MoE) deterministically assigns audio and video tokens their soodallit--seccffe FFNs/Experts. This design similar BAGEL (Deng al, 2025), which allocates understanding and generation tokens to separate FFNs, while we instead assign tokens based on modality. The advantage is that, after sufficient cross-modal interaction through attention, modality interference in FFN is isolated, allowing each branch to focus on intra- modal feature modeling. Similar to traditional MoE. benefits, although the total parameter size increases from 1.3B 2.1B, the number of activated parameters per token remains 1.3B. Thus, expand model capacity to improve performance without adding inference overhead.
3.3 TEMPORAL-ALIGNE ROTARY POSITION ENCODING
Establishing a shared temporal reference for audio and video tokens crucial for achieving gen- eration synchrony (Liu et al. 2025c; Wang et al.) (025c).. In contrast to the spatial-temporal prior (ST-Prior) employed in Javis (Liu et al. 2025c) and the frame-level cross-attention mechanism in Verser (Wang et al. 2025a). this paper proposes more direct and precise control strategy: an audio-visual Temporally Aligned Rotary Position Encoding (TA-RoPE). Specifically, absolute temporal aligmment is enforced along the first dimension (dimension 0) of the 3D position IDs for both audio and video tokens, as illustrated in Fig.
We first retain the 3D RoPE formulation for video tokens introduced in Wan2.1 (Wan et al. 2025). Given video tokens of shape T, x H x W, their 3D position IDs range from (0,0,0 0,0) to (T, H, W 1). For audio tokens of shape Ta M (extracted from meelsppectograms as in AudioLDM2 (Liu et al. (2024b)), we first augment them with an additional leading dimension (dimension 0) align with the video tokens along the absolute time axis. This means, audio tokens corresponding to the same time window of video tokens with temporal ID are assigned temporal ID i well. Subsequently, to ensure that audio and video position IDs remain strictly -oneolerl,,,,ng, we offset the original mel-spectrogram dimensions (T, and M) by adding H and W, respectively. Consequently, the audio position IDs span from (H, W) to (H + T. 1, W + M - 1). Further discussion and comparisons are provided in the Sec. C
Formally, for an audio token at timestamp t and frequency bin m, its positional ID is defined as:
(1+
T To
Ra(t, m)
t H.m H, +WW
(4)
where [-] denotes the round operation. This formulation explicitly enforces temporal alignment and synchrony between audio and video tokens.
Note that, thanks to the full attention design in Wan2.1, we can logically emulate the interleaved audio-video temporal arrangement shown in Fig. purely through position ID manipulation, without physically reordering tokens. In contrast, within a causal (autoregressive) DiT framework, achieving temporal alignment would require physically interleaving audio and video tokens such that those with smaller temporal position IDs are placed earlier in the sequence. However, such physical reordering entails non-contiguous memory accesses and incurs additional computational overhead.
5

---

Published as a conference paper at ICLR 2026
Wimning-Losing
Text Caption
AV-DPO
audio" valeo rest
mon helding o neto har black hird perched his The bird pecking the background.
audio valeo test
Referce Model
<Andio, Videa> Per Tou
Roward Medels
Scores
Andia Scare
Audio Quality
0.22
0.72 0:51
seme boxes stooked and coterful cinoulon object tiree sizeo.
Multimodal Score
TA-Align
0009
0.64 0.55]
Video Score
Visual Quality
0.800
Motion Quality
Real Vidro-Audie
TV-Alig
AV Score
AV-Align
[0:0] 0.366
AV-Sync
Figure 5: Illustration of preference data collection and training pipeline of audio-video DPO.
3.4 DIRECT PERFORMANCE OPTIMIZATION FOR JOINT AUDIO-V VIDEO GENERATION
To further improve the video and audio quality and synchronization, we propose an audio-video direct preference optimization (AV-DPO) algorithm to align JAVG models with human preferences. The overall pipeline is shown in Fig- 5 and our designed AV-DPO features some core contributions: (1) AV-DPO is the first one align rectified flows with preference data for joint audio-video gener- ation; (2) AV-DPO adopts various reward models to automatically and comprehensively evaluate the generated audio-video samples, and performs ranking based modality-aware dimensions to en- sure modality consistency in selected chosen-rejection pairs. The details are introduced as follows.
Reward Models. We employ various reward models to evaluate audio-video data from three modality-aware dimensions: audio reward, video reward, and audio-video alignment. Specifically, for audio reward, we leverage AudioBox (Tjandra et al., 2025) to comprehensively evaluate the au- dio quality, with ImageBind (Girdhar et al. 2023) to measure text-audio semantic similarity (align- ment). As for video reward, we take VideoAlign (Liu al.] (025b) to assess visual and motion quality, with ImageBind (Girdhar et al. (023) to measure text-video semantic alignment. For audio- video alignment, we also utilize ImageBind to calculate audio-video semantic similarity, with Sync- former (lashin et al., 2024) to estimate temporal synchrony. To solve the problem that different reward indicators yield various magnitudes. we first normalize the scores each metric and average them to obtain the reward for the corresponding modality- dimension.
Preference Data Acquisition. To construct the preference data, we first curate a prompt pool P with 30k text captions apart from the SFT training data, and then prompt the reference model to generate N = 3 audio-video pairs for each prompt. To stabilize the preference optimization, we also add the ground truth audio-video pairs into the generated ones to form the candidate preference data, which are evaluated by the previously mentioned reward models from three modality-speciff dimensions. Afterwards, we select the winner sample a." o yielding better ranking scores than the loser sample at, of across all dimensions:
) }}
de (a
du Sj of)
Du =
(a 2 ui a v/ wi) yy E Pt, Luefa,
(5)
where (*) represent the normalized score of the j-th reward of modality u. Finally, eq. (5) results in around 25k audio-video preference pairs. It is worth mentioning that winning pairs from our generated samples occupy around 30% of the total preference data, indicating that the baseline model itself already possesses fairly strong generative capabilities.
Direct Preference Optimization. Unlike prior single-modality DPO approaches (Liu et al. 2025b),
AV-DPO enhances joint audio-video generation by considering modality-awart preference:

---

Published as a conference paper at ICLR 2026
Table 1: Main results on JavisBench for generating 240p4s sounding videos. The best results are marked with bold, and the second ones are marked with underline.
AV-Quality
Text Consistency
AV-Consistency
AV-Synchrony
Model
Size
Runtime +
FVD +. FAD TV.IB 1 TA-IB CLIP CLAP
AB
AVIIScore
JavisScure
DeSync
1244A2V
TempoTkk
1.30
539.8
0.084
0.205
0.139
0.122
0.103
1.532
20s
TPOS
1.08
839.7
0.201
0.229
0.124
0.129
0.095
1.493
19s
T2V4V2A
0.104
1.071
SeaR 0.4B
7.6
0.129
0.263
0.160
0.143
0.112
1.099
25s
FoleyC
0.193
0.151
MMAudio
0.18
6.1
0. 160
0.407
0.198
0.182
0,150
0.849
15s
T2AV
MM-Diff
0.48
2311.9
27.5
0.080
0.014
0.181
0.079
0.119
0.109
0.070
0.875
96
JavisDiT
204.1
0.197
UniVerse-1
6.4B
194.2
8.7
0,272
0.111
0.309
0.245
0.104
0.098
0.077
0.929
13s
Ours
2,18
141.5
5.5
0.282
0.164
0.316
0.424
0.198
0.184
0.159
0.832
Diff policy
|l (x(" 2.30
v.w 112
vo(x," 2d v
113
Difffet
Vier Vxer(x, 2.00 .t) ".
12 Vier(x nll tt) v. 1
Diff policy
= 11 Va(X, a,w0 ,t) V.
112 - Vo a. .t) - aod 11a
Diffeer
= (xer(x, .t)
va,
112 || Vier( anl va. 112
CBO
and x
log of Biifff -Dfff Diff (a(iffffulic
policy Difff
AV-DPO promotes well-aligned outputs while suppressing misaligned ones using curated preference pairs, with flow matching loss added for regularization (Hung et al. (024) to avoid overfitting.
4
EXPERIMENTS
Sec. 4.1 and Sec. 4.2 present the setup and results of the main experiments, while Sec. 4.3 provides in-depth analysis of the key modules. More ablation studies are included in Sec. D
4.1 EXPERIMENTAL SETUP
Implementation Details. Our model is built upon Wan2.1-1.3B-1T (Wan et al., 2025) and pro- gressively adapted for joint audio-video generation: audio pre-training, audio-video SFT, and audio- video DPO. Rectified flow (Liu et al.) (023) is adopted as the noise scheduler for diffusion optimiza- details are provided in Sec. B.I
tion. All the
Training Datasets. For audio data, we directly adopt the 780K audio-text pairs collected by Jav- isDiT (Luu et al., 2025c) for audio pre-training. For video data, we filtered 330K audio-video-ten
triplets from TAVGBench for audio-video SFT, along with additional 25K samples for audio- video DPO. Further details and investigations are provided in Sec. B.2 and Sec. D.2
Evaluation Benchmarks. We mainly follow JavisDiT (Liu et al. 2025c) to conduct experiments JavisBench (10,140 140 samples for the main experiments) and Bavis-ench-min (1,000 samples for ab- lation studies). The evaluation covers 11 metrics across various audio-vide dimensions, including quality, consistency, and synchrony. Further details and explanations are provided in Sec. B.3
4.2 MAIN RESULTS
As shown in Tab. [ our JavisDiT++ model significantly outperforms previous JAVG methods in all dimensions of audio-video generation. In particular, we surpass Vnivers (Wang et al.) 2025a) by a large margin on quality and consistency, which also adopts Wan2.1-1.331 (Wan et al. (025) as the backbone. This improvement is attributed to the unified and efficient design of MS-MoE. rather than simply stitching two pretrained models together as in Uni UniVerse-L. I. On the other hand, our model also substantially outperforms both Jiv and Unrverse-1 on synchrony metrics, benefiting from

---

Published as a conference paper at ICLR 2026
turtle swims in turquoise water among small fish, with birds chirping the background.
Prompt:
Prompt: girl in white headscarf, black top, red skirt plays the flute beside another on piano.
Latency
Veo-3
Veo-3
JavisDiT
JavisDiT
3m55s
will
UniVerse-
UniVerse-1
1m42s
Ours
Ours
1m4s
[
Figure 6: Generations from recent JAVG models. Best viewed zoom or supplementary materials.
the explicitly designed TA-RoPE strategy that provides accurate audio-video temporal alignment. Fig. [6 further presents qualitative comparisons, where generated results consistently surpass JavisDiT and UniVerse-l, narrowing the gap to the advanced Veo3. In addition, Fig. 6 also states that our model introduces only 1.6% additional inference cost over the Wan2.1 backbone (Im3s), significantly more efficient than two-stream methods like Uni Verse-1 and JavisDiT.
4.3 IN-DEPTH ANALYSIS
In this section, we take JavisBench-nin (Liu et al. 2025c) with 1,000 prompts as the test set for evaluation. To avoid redundancy, we report out of the 11 full metrics of audio-video generation.
Table Investigation on architectural designs to adapt Sam2.1-T2V to joint audio-video generation.
Quality
Consistency
Synchrony
Arch Design
FVD
FAD
TV-IB 1
TA-IB 1
AV-IB
JavisScore t DeSync
Shared-DiT + LoRA
227.6
6.51
0.283
0.138
0.127
0.098
0.934
Shared-DiT + Full-FT
269.3
5.66
0.276
0.159
0.164
0.137
0.945
MS-MoE (Ours)
221.3
5.51
0.283
0.163
0.194
0.153
0.807
The proposed modality-peecific MoE is capable of JAVG. Tab. Lab.2oo compares three different strate- gies for performing audio pre-training and audio-video SFT within unified model. First, we reuse 1- as the shared DiT (Zhao et al., 2025) and apply either LoRA or full- parammeereer finetun- ing for torte-to--uuioo-vieee (T2AV) adaptation. which serve two important baselines. According to Tab. 2 the A scheme suffers from poor audio quality and consistency due to its limited trainable capacity; the full-finetuning scheme, on the other hand, shifts too many parameters during the audio pre-training stage, which severely degrades video quality and consistency. In contrast, MS-MoE design preserves strong video generation ability while equipping the model with high-quality audio generation, and simultaneously maintains excellent audio-vide synchrony. In addition, Sec. D.2 reveals that ensuring good data quality is the foundation to increase the sample quantity to improve training efficacy, providing new insight to scale up JAVG models in the future.
Our LoRA configuration a suitable choice. Fig. 7 illustrates the effects of different settings for T2AV adaptation, where "A-LoRA" and "A-noL LoRA" refer whether adding LoRA during
8

---

Published as a conference paper at ICLR 2026
A-LoRA AV-LoRA (1-64)
A-noLoRA AV-LoRA (164)
A-noLoRAA +VV-LORA (r122)
A-noLoRA + -VVAAtLLRAA
A-noLoRA AV-LoRA (1-32)
FVD 1
FAD
TV-IB 1
TA-IB
AV-IB ?
JavisScore [
DeSyne |
Figure Comprehensive ablation studies on LoRA configurations.
audio pretraining, "AV-AttnLoRA" and "AV-LoRA" refer to adding LoRA to the attention blocks or to the whole DiT during audio-video joint training, and "r" denotes the compression rank. Fig. 7 reports normalized scores for each evaluation metric to compare the relative differences, where we can draw several conclusions: (1) Adding LoRA to the attention layers during the audio pre- training stage (A-LoRA) cannot improve audio generation quality but significantly reduces video generation performance, as alters parameters in the video branch. (2) Compared with adding LoRA only the attention layers (AV-AttnLoRA), also applying LoRA FFN (AV-LoRA) leads to a notable improvement in audio-video generation performance, since the T2AV adaptation task remains relatively challenging. (3) The model performance is not highly sensitive the choice of LoRA rank and alpha parameters, empirically, setting rank = 64 brings slightly better performance.
The proposed TA-RoPE synchro- nization mechanism effective and efficient. Tab. compaaes several de- signs for audio-video generation syn- chrony. First, although both the ST- Prior proposed in Jiv (Liu et al.] 2025c) and the frame-level cross- attention used in UniVerse-1 1 (Wang jct al. 2025a) improve synchrony. they bring unaffordable increase in inference latency. In contrast, our
Table 3: Comparison of synchronization mechanisms.
Mechanism
JavisScore
DeSyne 4 Latency 4
None
0.142
0,942
1m4s
ST-Prior
0.145
0,863
Iml0s
FrameAttn
0.124
0.850
1m22s
TA-RoPE (Ours)
0.153
0.807
1m4s
TA-RoPE ST-Prror
0.155
0.856
1m10s
TA-RoPE + FrameAttn
0.151
0,802
1m22s
TA-RoPE strategy achieves better performance with zero additional inference cost. On the other hand, TA-RoPE can also be combined with ST-Prior or FrameAttn to yield slightly better synchrony, but we ultimately discard these combinations to maintain inference efficiency and overall simplicity. Due to space limitations, a systematic investigation of TA -A-RoP is placed at Sec. D.I
The AV-DPO design is reasonable and effective. As Sec. D.3 validates the dyper-parameds (in- cluding f and learning rate) choices, Tab. 4 mainly compares different reward strategies for se- lecting win-lose pairs DPO training. First, applying modality aagnoity-agnostic -gnostic strategies like Average- Micro (averaging across all metrics before ranking (Liu et al. 2025b)) and Average-Macro (ranking within each metric and then averaging (Xue al., 2025)) fails achieve consistent improvements in audio-video generation. This is because they may form a win sample by combining better video but worse audio, which conflicts with eq. (6). In contrast, calculating rewards separately for audio and video and ensuring modality-consistee improves generation quality, consistency, and synchrony. Meanwhile, removing normalization (i.e., w/o norm) reduces the accuracy of pair ranking due to scale and range differences across rewards, which in turn degrades DPO performance. Likewise, discarding ground-truth samples and forming pairs only from generated candidates (i.e., w/o gt) even gets worse results, as differences among generated samples are often too small to guide preference shifts.
chosen pairs (e.g.. Modality-Micrt Micro/--oio////aroo/) effectively
4.4 HUMAN EVALUATION
In this section, we further conduct user studies to more comprehensively evaluate the performance of joint audio-video generation through human evaluation. Specifically, reuse the 100 prompts

---

Published as a conference paper at ICLR 2026
Table 4: Investigation on the effectiveness of different AV-DPO reward strategies.
Quality
Consistency
Synchrony
Reward Design
FVD 4
FAD 4
TV-IB t
TA-IB t
AV-IB
JavisScore
DeSyne +
None (baseline)
221.3
5.51
0.283
0.163
0.194
0.153
0.807
Average-Micro
199.7
5.28
0.281
0.166
0.199
0.154
0.810
Avcrage-Macro
203.3
5.31
0.281
0.166
0.196
0.152
0.810
Modality-Miero
198.5
5.32
0.284
0.168
0,201
0.156
0.776
Modality-Mal -acro
201.1
5,41
0.282
0.166
0.197
0.156
0.773
Modality-Micnt (w/o norm)
210.0
5.34
0.281
0.167
0.197
0.153
0.821
-odcl (w/o gt)
234.7
5.43
0.281
0.164
0.197
0,154
0.833
from Fig. 2 and ask different models to generate corresponding audio-video outputs. For each
prompt, we randomly sample the outputs of two models and recruit three volunteers perform blind win-tie-loss preference judgments. The averaged scores across annotators are used as the final evaluation metric.
Comparison with baseline methods
Win
Tie
Lose
Ours vs Jiv
74.0%
25.3%
Ours Universe-1
74.7%
24.0%
20
60
80
100
Pereentage (%)
Figure Subjective comparison on generation with baseline methods.
Evaluation on AV-DPO efficacy
Win
Tie
Lose
SFT
20
40
60
100
Percentage (%)
Figure 9: Subjective evaluation on the efficacy of the AV-DPO strategy.
Our model substantially outperforms the baselines. Fig. 8 shows that our method surpasses
JavisDiT (Liu et al., (025c) and UniVerse-1 (Wang et al., 2025a) by more than 70% in human preference. These subjective results align well with the objective evaluations in Tab. T jointly validating the effectiveness of our approach.
The AV-DPO strategy effectively improves generation preference. Although the gains from DPO were relatively modest in the previous objective evaluations, Fig. 9 demonstrates that the DPO- enhanced model produces over 25% more videos favored by human annotators, which further sup- ports our motivation.
CONCLUSION
This work presents JavisDiT++, concise and efficient framework for native joint audio-video gen- eration. By introducing the MS-MoE design for modality-specific quality enhancement, the TA- RoPE strategy for explicit temporal alignment, and the AV-DPO algorithm for preference alignment, our model achieves state-of-the-a performance in quality, consistency, and synchrony. Built upon Wan2.1-1.3B-- --V and trained with only 1M data entries, our JavisDiT++ significantly outperforms existing open-source approaches while maintaining efficiency. We believe this work can establish an important milestone for JAVG and open new directions for future research in the field.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Kevin Black, Michael Janner, Yilun Du, llya Kostrikov, and Sergey Levine. Training diffusion models with reinforcement learning. arXiv preprint arXiv:2305. /3301, 2023.
Weilin Cai, Juyong Jiang. Fan Wang, Jing Tang, Sunghun Kim, and Jiayi Huang. A survey on mixture of experts large language models. IEEE Transactions on Knowledge and Data Engi- neering, 2025.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6308, 2017.
Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio- visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 721-725. IEEE, 2020.
Huakang Chen, Yuepeng Jiang, Guobin Ma. Chunbo Hao, Shuai Wang, Jixun Yao, Ziqian Ning, Meng Meng, Jian Luan, and Lei Xie. Diffrhythm++: Controllable and flexible full-length song generation with preference optimization. arXiv preprint arXiv:2507.12890, 2025.
Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, and Yuki Mit- sufuji. Mmaudio: Taming multimodal joint training for high-quality video-to-audio synthesis. In Proceedings of the Computer Vision and Puttern Recognition Conference, pp. 28901-28911 2025.
Hyung Won Chung, Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, and Orhan Firat. Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining, arXiv preprint vrXiv:2344.09557 2023.
Meihua Dang, Anikait Singh, Linqi Zhou, Stefano Ermon, and Jiaming Song. Personalized prefer- ence fine-tuning of diffusion models. In Proceedings of the Computer Vision and Pattern Recog- nition Conference, pp. 8020-8030, 2025.
Google DeepMind. Veo 3: Tech report, 2025. URL www.:/miiddddmmaeeeeeeeeeeeeeeee 3- Tech- Report ppf
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie. Ziang Song, et al. Emerging properties in unified multimodal pretraining. arXiv preprint arXiv:2505. /4683, 2025.
Konstantinos Drossos, Samuel Lipping. and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Process- ing (ICASSP), pp. 736-740. IEEE, 2020.
Ying Fan, Olivia Watkins, Yuqing Du, Hao Liu, Moonkyung Ryu, Craig Boutilier, Pieter Abbeel, Mohammad Ghavamzadeh, Kangwook Lee, and Kimin Lee. Dpok: Reinforcement learning for fine-tuning text-to-image diffusion models. Advances in Neural Information Processing Systems, 36:79858-7988 79885, 2023.
Hiroki Furuta, Heiga Zen, Dale Schuurmans, Aleksandra Faust, Yutaka Matsuo, Percy Liang, and Sherry Yang. Improving dynamic object interactions in text-to-video generation with ai feedback. arXiv preprint arXiv:2412.02617, 2024.
Xiaoxue Gao, Chen Zhang, Yiming Chen, Huayun Zhang, and Nancy F Chen. Emo-dpo: Control- lable emotional speech synthesis through direct preference optimization. In ICASSP 2025-2025 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2025a.
Xin Gao, Li Hu, Siqi Hu, Mingyang Huang, Chaonan Ji. Dechao Meng, Jinwei Qi, Penchong Qiao, Zhen Shen, Yafei Song. et al. Wan-s2y: Audio-drived cinematic video generation. arXiv preprint arXiv:2508. 18621, 2025b.
11

---

Published as a conference paper at ICLR 2026
Zhifu Gao, Zerui Li, Jiaming Wang, Haoneng Luo, Xian Shi, Mengzhe Chen, Yabin Li, Lingyun Zuo, Zhihao Du, Zhangyu Xiao, and Shiliang Zhang. Funasr: A fundamental end-to-end speech recognition toolkit. In INTERSPEECH, 2023.
Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore. Manoj Plakal, and Marvin Ritter. Audio set: An ontology and audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 776-780. IEEE, 2017.
dataset for
Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 15180-15190, 2023.
Junmin Gong, Sean Zhao, Sen Wang, Shengyuan Xu, and Joe Guo. Ace-step: A step towards music generation foundation model. arXiv preprint 20li:25000000045, 2025.
Kaisi Guan, Xihua Wang. Zhengfeng Lai, Xin Cheng, Peng Zhang, XiaoJiang Liu, Ruihua Song, and Meng Cao. Taming text-io-sounding video generation via advanced modality condition and interaction. arXiv preprini arXiv:2510.03111 2025.
Andrey Guzhov, Federico Raue, Jörn Hees, and Andreas Dengel. Audioclip: image, text and audio. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 976-980, IEEE, 2022.
Extending clip to
Yoav HaCohen, Benny Brazowski, Nisan Chiprut, Yaki Bitterman, Andrew Kvochko, Avishai Berkowitz, Daniel Shalem, Daphna Lifschitz, Dudu Moshe, Eitan Porat, et al. Lix-2: Efficient joint audio-visual foundation model. arXiv preprint arXiv:2001.0223: 2026.
Akio Hayakawa, Masato Ishii, Takashi Shibuya, and Yuki Mitsufuji. Mmdisco: Multi-modal discriminator-guided cooperative diffusion for joint audio and video generation. arXiv preprint arXiv:2405.17842 2024.
Teng Hu, Zhentao Yu, Guozhen Zhang, Zihan Su, Zhengguang Zhou, Youliang Zhang, Yuan Zhou, Qinglin Lu, and Ran Yi. Harmony: Harmonizing audio and video generation through cross-task synergy, arXiv preprint arXiv:2511.21577 2025.
Qihan Huang, Long Chan, Jinlong Liu, Wanggui He, Hao Jiang, Mingli Song, and Jie Song- Patchdpo: Patch-level dpo for finetuning- free personalized image generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18369-18378, 2025a.
Xiaohu Huang, Hao Zhou, Qiangpeng Yang, Shilei Wen, and Kai Han. Jova: Unified multimodal learning for joint vidco-audio generation. arXiv preprint arXiv:2512.13677, 2025b.
Chia-Yo Hung, Navonil Majumder, Zhifeng Kong, Ambuj Mehrish, Amir Ali Bagherzadeh, Chuan Li, Rafael Valle. Bryan Catanzaro, and Soujanya Poria. Tangoflux: Super fast and faithful text to audio generation with flow matching and clap-ranked preference optimization. arXiv preprint arXiv:2412.21037, 2024.
Vladimir Iashin, Weidi Xie, Esa Rahtu, and Andrew Zisserman. Synchformer: nization from sparse cues. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.
Efficient synchro-
IEEE, 2024.
Yujin Jeong, Wonjeong Ryoo, Seunghyun Lee, Dabin Sco, Wonmin Bycon, Sangpil Kim, and Jinkyu Kim. The power of sound (tpos): Audio reactive video generation with stable diffusion. In Proceedings of the IEEE/VE International Conference on Computer Vision, pp. 7822-7832, 2023.
Yujin Jeong, Yunji Kim, Sanghyuk Chun, and Jiyoung Lee. Read, watch and scream! sound gener- ation from text and video. arXiv preprint arXiv:2407.0555/ 2024.
Yuxuan Jiang, Zehua Chen, Zeqian Ju, Chang Li, Weibei Dou, and Jun Zhu. Freeaudio: Training- free timing planning for controllable lorg-form text-to-audio generation. arXiv preprint arXiv:2507.085577 2025.
12

---

Published as a conference paper at ICLR 2026
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Vol- ume 1 (Long and Short Papers), pp. 119-132, 2019.
Seung Hyun Lee, Gyeongrok Oh, Wonmin Byeon, Chanyoung Kim, Won Jeong Ryoo, Sang Ho
Yoon, Hyunjun Cho, Jihyun Bae, Jinkyu Kim, and Sangpil Kim. Sound-guided semantic video generation. In European Conference on Computer Vision, pp. 34-50. Springer, 2022.
Ruilong Li, Shan Yang, David A Ross, and Angjoo Kanazawa. Ai choreographer: Music condi- tioned 3d dance generation with aist++. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 13401-13412, 2021.
Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai. Real-time scene text detection with differentiable binarization. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 1 1144-11441, 2020.
Haohe Liu, Gael Le I Lan, Xinhao Mei, Zhaoheng Ni, Anurag Kumar, Varun Nagaraja, Wenwu Wang, Mark Plumbley, Yangyang Shi, and Vikas Chandra. Syncflow: Toward temporally aligned joint audio-video generation from text. arXiv preprint arXiv:2412.15220 2024a.
Haohe Liu, Yi Yuan, Xubo Liu, Xinhao Mei, Qiuqiang Kong, Qiao Tian, Yuping Wang, Wenwu Wang, Yuxuan Wang, and Mark D Plumbley. Audioldm 2: Learning holistic audio generation with self superviie pretraining. IEEE/ACM Transactions on Audio, Speech, and Language Pro- cessing, 2024b.
Jie Liu, Gongye Liu, Jiajun Liang, Yangguang Li, Jiaheng Liu, Xintao Wang, Pengfei Wan, Di Zhang, and Wanli Ouyang. Flow-grpo: Training flow matching models via online rl. arXiv preprint arXiv:2505. 0055400 2025a.
Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Wenyu Qin, Menghan Xia, et al. Improving video generation with human feedback. arXiv preprint 250.:220011991111 2025b.
Kai Liu, Wei Li, Lai Chen, Shengqiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Rongxin Jiang, Jiebo Luo, Hao Fei, et al. Javisdit: Joint audio-video diffusion transformer with hierarchical spatio-temporal prior synchronization. arXiv preprint arXiv:2503.23377 2025c.
Runtao Liu, Haoyu Wu, Ziqiang Zheng, Chen Wei, Yingqing He, Renjie Pi, and Qifeng Chen. Videodpo: Onni-preference alignment for video diffusion generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8009-8019, 2025d.
Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In The Eleventh International Conference Learning Repre- sentations (ICLR), 2023.
Chetwin Low, Weimin Wang, and Calder Katyal. Ovi: Twin backbone cross-modal fusion for audio- video generation. arXiv preprint arXiv:2510.01284 2025.
Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized video-to-audio
synthesis with latent diffusion models. Advances in Neural Information Processing Systems, 36: 48855 48876, 2023.
Jayy Lyu, Xing Lan, Guohong Hu, Hanyu Jiang, Wei Gan, and Jian Xue. Etau: Towards emo- tional talking head generation via facial action unit. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pp. 1-6, 2024. doi: 10.1 1099/CCEE5554.2222...0666
Jayy Lyu, Xing Lan, Guohong Hu, Hanyu Jiang, Wei Gan, Jinbao Wang, and Jian Xue. Multimodal emotional talking face generation based on action units. IEEE Transactions on Circuits and Systems for Video Technology, 35(5):4026 400044444488888. 2025. doi: 10. 101100/SST.22243533355555
Yuxin Mao, Xuyang Shen, Jing Zhang, Zhen Qin, Jinxing Zhou, Mochu Xiang, Yiran Zhong, and Yuchao Dai. Tavgbench: Benchmarking text to audible-vides generation. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 6607-6616, 2024.
13

---

Published as a conference paper at ICLR 2026
Irene Martin-Morató and Annamaria Mesaros. What is the ground truth? reliability of multi- annotator data for audio tagging. In 2021 29th European Signal Processing Conference (EU- SIPCO), pp. 76-80 80. IEEE, 2021.
Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumb- ley, Yuexian Zou, and Wenwu Wang. Waveaps: A shatgpt-assisted weakly- audio caption- ing dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
Karol Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pp. 1015-1018, 2015.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in neural information processing systems, 36:53728-5331
2023.
Ludan Ruan, Yiyang Ma, Huan Yang, Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10219-10228, 2023.
Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pp. 1041- 1044, 2014.
Christoph Schuhmann. Improved aesthetic predictor, March 2022. URL httpp:///gthubb com/
Bob L Sturm. The gtzan dataset: Its contents, its faults, their effects on evaluation, and its future use. arXiv preprint arXiv: /306. /461, 2013.
Mingzhen Sun, Weining Wang. Yanyuan Qiao, Jiahui Sun, Zihan Qin, Longteng Guo, Xinxin Zhu, and Jing Liu. Mm-ldm: Multi-modal latent diffusion model for sounding video generation. In Proceedings of the 32nd ACM International Conference on Multimedia, pp. 10853-10861, 2024.
Zineng Tang, Ziyi Yang, Chenguang Zhu, Michael Zeng, and Mohit Bansal. Any-to-any generation via composable diffusion. Advances in Neural Information Processing Systems, 1083-1199,,,, 2023.
Zineng Tang, Ziyi Yang. Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal. Codi- 2: In-context interleaved and interactive any-to-any generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 27425-27434, 2024.
Zeyue Tian, Yizhu Jin, Zhaoyang Liu, Ruibin Yuan, Xu Tan, Qifeng Chen, Wei Xue, and Yike Guo. Audiox: Diffusion transformer for anothing-to-audio generation. arXiv preprint arXiv:2503. 10522, 2025.
Andros Tjandra, Yi-Chiao Wu, Baishan Guo, John Hoffman, Brian Ellis, Apoorv Vyas, Bowen Shi, Sanyuan Chen, Matt Le, Nick Zacharov, et al. Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound. arXiv preprint arXiv:2502.05139, 2025.
Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, and Nikhil Naik. Difffusion model alignment using direct preference optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8228 $228-8238, 2024.
Team Wan, Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen- Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang. et al. Wan: Open and advanced large-scale video generative models. arXiv preprint 2.XX::2503.20114 2025.
14

---

Published as a conference paper at ICLR 2026
Duomin Wang, Wei Zuo, Aojie Li, Ling-Hao Chen, Xinyao Liao, Deyu Zhou, Zixin Yin, Xili Dai, Daxin Jiang, and Gang Yu. Universe-1: Unified audio-video generation via stitching of experts. arXiv preprint ar arXhv:2509.06155, 2025a.
Junke Wang. Zhi Tian, Xun Wang. Xinyu Zhang. Weilin Huang, Zuxuan Wu, and Yu-Gang Jiang. Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl. arXiv preprint ar0iv:2504.1 1455, 2025b.
Kai Wang, Shijian Deng, Jing Shi. Dimitrios Hatzinakos, and Yapeng Tian. Av-dit: Effi- cient audio-visual diffusion transformer for joint audio and video generation. arXiv preprint arXiv:2406. 07686, 2024.
Le Wang, Jun Wang. Chunyu Qiang. Feng Deng, Chen Zhang, Di Zhang, and Kun Gai. Audiogen- omni: A unified multimodal diffusion transformer for audo-ssynchronized audio, speech, and song generation. arXiv preprint arXiv:2508.007.3 2025c.
Yusong Wu*, Ke Chen*, Tianyu Zhang* Yuchen Hui*, Taylor Berg- Kink-KKrpattrick and Shlomo Dubnov. Large-scale contrastive language-audio pretraining with feature fusion and keyword- to-caption augmentation. In IEEE International Conference Acoustics, Speech and Signal Processing, ICASSP, 2023.
Ziyi Wu, Anil Kag, Ivan Skorokhodov, Willi Menapace, Ashkan Mirzaei, Igor Gilitschenski, Sergey Tulyakov, and Aliaksandr Siarohin. Densedpo: Fine-grained temporal preference optimization for video diffusion models. arXiv preprint 2xXxv22506.03117, 2025.
Yazhou Xing, Yingqing He, Zeyue Tian, Xintao Wang, and Qifeng Chen. Seeing and hearing: Open- domain visual-audio generation with diffusion latent aligners. In Proceedings of the IÊEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7151-7161, 2024.
Haofei Xu, Jing Zhang, Jianfei Cai, Hamid Rezatofighi, Fisher Yu, Dacheng Tao, and Andreas Geiger. Unifying flow, stereo and depth estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(11):139)) 11:13911-13958 2023.
Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, al. Qwen2. 5-omni technical report. arXiv preprint arXiv:2503.20215. 2025.
Zeyue Xue, Jie Wu, Yu Gao, Fangyuan Kong, Lingting Zhu, Mengzhao Chen, Zhiheng Liu, Wei Liu, Qiushan Guo, Weilin Huang, et al. Dancegrpo: Unleashing grpo on visual generation. arXiv preprint arXiv:2505.07811 2025.
Kai Yang. Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Weihan Shen, Xiaolong Zhu, and Xiu Li. Using human feedback to fine-tune diffusion models without any reward model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8941-8951, 2024.
Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi. Diverse and aligned audio-to-videe generation via text-to-video model adaptation. In Proceedings of the AAAI Con- ference on Artificial Intelligence, volume 38, pp. 6639-6647, 2024a.
Guy Yariv, Itai Gat, Sagie Benaim, Lior Wolf, Idan Schwartz, and Yossi Adi. Diverse and aligned audio-to-video generation via text-to-video model adaptation. In Proceedings of the AAAI Con- ference Artificial Intelligence, volume 38, pp. 6639-6647, 2024b.
Shihao Yuan, Yahui Liu, Yang Yue, Jingyuan Zhang, Wangmeng Zuo, Qi Wang, Fuzheng Zhang, and Guorui Zhou. Ar-grpo: Training autoregressive image generation models via reinforcement learning. arXiv preprint arXiv:2508.0697 2025.
Hui Zhang, Zuxuan Wu, Zhen Xing, Jie Shao, and Yu-Gang Jiang. Adadiff: Adaptive step selection for fast diffusion. arXiv preprint 2r3iv:231 14768, 2023.
Yiming Zhang, Yicheng Gu, Yanhong Zeng, Zhening Xing, Yuancheng Wang, Zhizheng Wu, and Kai Chen. Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds. arXiv preprint (0t::2407.014944 2024.
15

---

Published as a conference paper at ICLR 2026
Lei Zhao, Linfeng Feng, Dongxu Ge, Rujin Chen, Fangqiu Yi, Chi Zhang, Xiao-Lei Zhang, and Xuelong Li. Uniform: A unified multi-task diffusion transformer for audio-video generation. arXiv preprint arXiv:2502.03887 2025.
Zangwei Zheng, Xiangyu Peng. Tianji Yang. Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open-sora: Democratizing efficient video production for all, March 2024. URL URLhttps: /qithub. com//ppaaitechh/pee--0000
16

---

Published as a conference paper at ICLR 2026
A DISCUSSION
A. POTENTIAL LIMITATIONS
While our framework demonstrates state-of-the-art performance in joint audio-video generation, several limitations remain and open promising directions for future work:
Training Data Scale. Our model is trained roughly 1M entries, which, although efficient, may constrain scalability compared with larger proprietary systems. Expanding the dataset with more diverse and high-quality audio-video pairs could further improve generalization and robustness.
Model Size. We adopt 1.3B-parameter backbone with crrameter-efficien adaptations. Scal- ing up to larger backbones may unlock stronger representational capacity, especially for captur- ing subtle temporal and semantic correlations across modalities.
Tull-Parameter Training. Our approach relies on parameter-efficient tuning (e.g., LoRA). Exploring full-parameter finetuning could provide additional performance gains, albeit with higher computational cost.
Controllable Generation.
Current experiments focus on general text-to-audio-video genera- tion. Extending controllability to domains such as music or speech, with fine-grained control over rhythm, pitch, timbre, or lexical content, is an important next step.
Unified Cross-Modal Generation. Beyond text-conditioned JAVG, broader tasks such as audio-to-vided (A2V), video-to-audio (V2A), and mudio-image-to-vidd (AI2V) offer oppor- tunities for unified multimodal generative framework. Developing models that seamlessly perform across these modalities would mark a significant milestone toward general-purpose audio-visual content generation.
We hope these directions inspire future research toward building more scalable, controllable, and unified multimodal generative systems.
A.2 ETHIC STATEMENT
All datasets and models used in this work are publicly available on the internet and do not involve any private or sensitive information. addition, part of the DPO data we plan release is generated by models themselves, ensuring that no personal privacy is infringed.
A.3 REPRODUCIBILITY STATEMENT
We provide detailed descriptions of model design, training, and evaluation in both the main paper and the appendix. Furthermore, all code, pretrained checkpoints, and processed datasets will be publicly released to ensure full reproducibility of our results.
A.4 LLM USAGE STATEMENT
Large Language Models (LLMs) were used solely as writing assistants, including tasks such as language polishing and presentation refinement. They were not involved in the conception core ideas or designs.
B DETAILED IMPLEMENTATIONS
B. MODEL DETAILS
Audio VAE, We reuse and freeze the audio encoder and decoder from AudioLDM2 (Liu et al.) 20246). All ID audio signals are resampled to 16 kHz and converted into 64-bin mel-spectrograms using window size of 64 ms and hop size of 10 ms. The spectrograms via VAE into 8-channel audio embeddings. To further reduce the token count, we apply a 2x2 patchify operation before feeding the tokens into the DiT for diffusion and denoising.
then 8 compressed
17

---

Published as a conference paper at ICLR 2026
Stage
Stage 2/ Stage
Self Atlention
Self Atlention
Cross Attention
Cross Attention
Video
Audio FFN
Video FFN
Audio FFN
FFN
Init
Frezze
Full Finetuning
LoRA Finehuning
Figure A1: Illustration of trainable parameters at different stages.
Table Al: Detailed settings for the three-stage training pipeline.
Setting
Stage
Stage
Stage 3
training purpose
Audio PreTrain
Audio Video SFT
Audio-Vides DPO
trainable modules
Audio FFN/Embedde/HHcad
LoRA
LoRA
trainable params
794M
121M
121M
learning rate
le-4
le-4
le-5
warm-up steps
1000
1000
100
weight decay
0.0
0.0
0.0
training samples
780K
330K
25K
resolution
dynamic
dynamic
duration
dynamic
dynamic dynamic
dynamic
batch size
dynamic
dynamic
epoch
50
2
GPU days (H100)
Video VAE. We reuse and freeze the video VAE from Wan2.1 (Wa et al. 2025 Except for the first frame, all subsequent video frames are temporally compressed by a factor of 4, while every frame is spatially compressed by 8x8, resulting in 16-channel video embeddings. To ensure further compactness, these video embeddings are also compressed with a 2x2 spatial patchify operation before being fed into the DiT.
Text Encoder. We also reuse and freeze Wan2.1's umT5-xxl (Chung et al. 2023) as the text- encoder, whose context length is 512.
Backbone DIT. Our model built on the powerful W.n2.1-1.3B-1 T2V (Wan et al. 2025 model, and progressively extends from text-to-video generation text-conditioned joint audio-video gen- eration. The 1- base model has 30 layers with a hidden dimension of 1536. We keep the original parameters frozen throughout the entire training process, updating only the newly troduced audio FFN (along with audio embedder, audio head, etc) and the LoRA components at different stages. The final model has only 2.1B parameters after merging LoRA components. De- tailed training settings are presented as follows.
B.2 TRAINING DETAILS
Details of the three-stage training pipeline. Our model progressively extends from 1-n2.1-1.3BB
T2V Wan et al., 2025 to JAVG through three stages: (1) Audio tre-Training, where the Audio FEN is trained on 780K audio-text pairs for 50 epochs with learning rate of lc-4; (2) Audio-Video SFT, where LoRA is applied to train on 330K audio-video-ter triplets for epoch with a learning rate of le-4; and (3) Audio-Video Video DPO, where the LoRA parameters are retained and further trained
18

---

Published as a conference paper at ICLR 2026
WavCaps (42.46%)
HQ Vidcos
SFT
Clean Videos
Videos
MMTrail (15.19%)
AudioSet (14.17%)
Scoring Filters
VGGSouun (20.37%)
AudCaps (5.52%)
Speech Videos
Figure A2: (Left): Diversified audio-text sources. (Right): Data filtering process from TAVGBench.
on 25K entries for epoch with learning rate of le-5, The specific trainable modules and other
training settings can be found in Fig. [T and Tab. A1
Details of the audio-video training data. Fig. A2 and filtering procedure. For audio data, we directly adopt 780K audio-text pairs from Javis-
demonstrates the detailed data composition
DiT (Liu al. 2025c) for audio pretraining, which covers various public datasets, including Au- dioSet (Gemmeke et al., 2017), AudioCaps (Kim et al. 2019). VGGSound (Chen et al. 2020),
WavCaps (Mei al. 20024), Clotho (Drossos al. 2020), ESC50 (Piczak 2015), GTZAN (Sturm
2013). MACS (Martín-Morató & Mesaros] 2021), and UrbanSound8K (Salamon et al.
2014). We
apply no data filtering strategy ensure maximal text-to-audio generation capability spanning gen-
eral sound, music, and speech.
For video data, we adopt a subset of 1.1 million text-video-audio triplets from TAVGBench (Mao et al. (024)) and conduct a series of filtering strategies. We first eliminate a large part of videos containing human speech by using the FunASR (Gao al. 2023) detection tool, and then follow OpenSora (Zheng et al. (024]) to filter out the videos with relatively lower quality through aesthetic scoring (Schuhmann) 2022). flow (motion) scoring (Xu et al. (023), and OCR scoring (Liao et al.) 2020). In the final filtered pool, 330K data entries are divided for audio-video SFT, and the other 25K samples are used for audio-video DPO to avoid overlapping. We also provide empirical evaluations on the data diversity and quality in Sec. D.2
B.3 EVALUATION DETAILS
Evaluation Setup. We mainly follow JavisDiT (Liu et al. 2025c) to evaluate the models Jav- isBench, which consists of 10,140 prompts for joint audio-video generation in various real-world scenarios. In ablation studies, we take the JavisBench-mini for fast evaluation, where the 1 1,000 prompts are randomly selected from the total 10,140 samples from JavisBench. All models are required to generate 240P, 4-second sounding videos for quantitative evaluation.
JavisBench provides a comprehensive evaluation of quality, consistency, and synchrony for audio- video generation results. Here, we briefly introduce the mechanisms of each evaluation dimension:
Audio / Video Quality: measuring the perceptual quality of the generated audio and video, including (1) Fréchet Video Distance (FVD): FVD = 1111111113++ + +2 2)) where (pr. E,) and (My. Ey) are the mean and covariance of ground -hound-truth truth and generated video features extracted by a pretrained I3D encoder (Carreira & Zisserman 2017). Lower is better, indicating the generated video distribution is closer to the real one; (2) Kernel Video Distance (KVD): similar FVD, but estimates distribution differences via a kernel-based method (Kernel Inception Distance style), which is more stable on smaller datasets; lower is better, and (3) Fréchet Audio Distance (FAD): same concept as FVD, but computed on audio features extracted by a pretrained AudioClip model (Guzhov et al. 2022), measuring distribution distance between generated and real audio; lower is better.
Text Consistency:
evaluating how well the generated audio and video semantically match the input text description, including (1) ImageBind (Girdhar et al. 2023) text-video cosine simi- larity: sim(t, v)
[raze: (1) (video(v)
(2) ImageBind text-audio cosine similarity: same pro-
[eeae (f)] -01l-11laaa [[ [V)]
19

---

Published as a conference paper at ICLR 2026
cess but with the audio encoder faudio: (3) CLIP-Score using CLIP (Radford et al. 2021 to compute semantic similarity between text and video (video frames are sampled, encoded, and averaged); and (4) CLAP-Scoree using CLAP (Wu* et al. 2023) to compute semantic similarity hetween text and audio.
Audio-Video Semantic Consistency: measuring the semantic alignment between generated audio and generated video, including (1) ImageBind audio-video cosine similarity, encoding both modalities into the same space and computing cosine similarity between video and audio features; and (2) Audio- Visual Harmony Score (AVHScore): introduced in TAVGBench (Mao et al. (024) as a way to quantify how well the generated audio and video align semanti- cally in shared embedding space. It is defined by computing the cosine similarity be- tween each video frame and the entire audio, then averaging across all frames: AVHScore = 1 Ex= cos (osammeee) (a) A higher AVHScore indicates stronger audio-video se- mantic consistency. Note that we remove the CAVP-Score (Luo et al. 2023) used in Jav- isDiT (Liu et al.] (025c) because this metric keeps a range from 0.798 to 0.801 and cannot capture the difference when evaluating semantic consistency.
N
evaluating spatiotemporal alignment in gener- ated audio-video pairs, including (I) JavisScore: a new metric proposed in JavisDiT (Liu jt al. 2025c). The core idea is to use a sliding window along the temporal axis to split the audio-video pair into short segments. For each segment, compute cross-modal similarity with ImageBind and take the mean score: JavisScore = */4 Ex= a(a,, u.), ous, ai)
Audio-Video Spatio-Temporal Synchrony:
x Σラ= top-ki (cos ( (v..g) (a:))); and (2) DeSyic: a metric adapted from Synch- min
former[lashin et al. 2024), which measures fine-grained temporal misalignment between audio and video streams. Specifically. it estimates the temporal offset of audio-visual events by per- forming 21 -category classification task (predicting the offset/asynchrony degree ranging from - 10 to 10 and taking the absolute values). A lower DeSyne score indicates better synchroniza- tion.
Compared Methods Since open-source models that support text-conditioned joint audio-video (JAVG) generation are still very limited, we follow JavisDiT (L.iu et al.
(025c) and include cascaded generation pipelines (i.e., T2A+A2V (Yariv et al. [0024a] Jeong et al.
2023) and T2V+V2A (Jeong al. 2024 Xing et al. 2024, Zhang al. [024]] Cheng al. 2025)) for comparison. In particular, AudioLDM2 (Liu et al. 2024b) is adopted to conduct the preliminary T2A generation for subsequent T2A+A2V evaluation, while OpenSora-v1.2 (Zheng et al. 2024) is used perform the preliminary T2V task for subsequent T2V+V2A methods. For JAVG-capable models, we treat the unconditional MMDiffusion (Ruan et al.) 2023) simple baseline, while focusing on comparisons against text- conditional models like Ji (Liu et al. 2025c) and 1niVerse (Wang et al. (002a)). For JavisDiT, we directly downloaded its released checkpoints and performed JavisBench generation locally, enabling us to compute the newly introduced DeSync (Iashin et al. (024) metric and update other metrics consistently. For 1niVerse-l, since it essentially supports audio-video generation from Text+ + Reference Image, we use the first frame of our model's 's generated video as the reference image, allowing UniVerse-1 to perform mndi-syyncroonized image animation.
Details of Fig. [ In Sec. [ we present radar chart (Fig. 2 to illustrate the performance differences among various JAVG models. Specifically,, we randomly sample 100 prompts from JavisBench (Liu et al. 2025c) for audio-video generation. For open-source models, we run inference locally, while for Veo3 (DeepMind] 2025), we obtain results via its API. After collecting outputs from all models, we follow the DPO reward setup in Sec. 3.4 comprchensively evaluate the performance, including (I) VideoAlign (Liu et al. 2025b) and AudioBox (Tjandra et al.] 2025) to assess video and audio quality; (2) ImageBind (Girdhar et al. 2023) to compute semantic similarity across TV-Align, TA- Align, and AV-Align; and (3) SynchFormer (lashin et (0024) to compute DeSync as the synchrony metric. Since DeSyne is defined as a "lower-is-better" score, whereas all other metrics are "higher-
is-better", we invert DeSync values in Fig. 2 to ensure visual consistency.
C DETAILED COMPARISON WITH RELATED WORKS
Architectural Difference with Recent JAVG Mudels. Fig. A3 presents a comparison between model design and recent JAVG approaches. First, UniForm (Zhao al. 2025) attempts to use
20

---

Published as a conference paper at ICLR 2026
Video Audio
Video
Audio
Video
Audio
Vidco
Audio
text
text si-prior
Attn
Self Attn
Setf Attn
Solf Attn
Self Actn
Solf
Cross Atta
Cross Attn
Cross Attn
Cross Attn
Cross Artn
Cross Attn
Cross
F
Bi-CA
Videe FFN
Audio FFN
FFN
FFN
FFN
FFN
Video
Audio
Video
Audio
Video
Audio
Video
Audio
(A) UniForm
(R) JavisDiT
(C) UniVerse-]
(D)
Figure A3: Architectural comparison with Uniform, JavisDiT, and Uni Verse-1.
Frame
Frame
Frame 2
600.9
(20.9
611140
W11.2
611,9
(1140
0.1,1
0.1.3
111
(2111)
(21,2
(11
602/0
(0221)
0U220
402,39
(1.2/
0.2.0
(1.22)
(12.3)
022/
(22211
(222
(22.0
00:3
(04/0)
(155
(16.0
47700
(88
(A) Ours
maa
000
0V0.3
250D
2502
2503
60:00
(000
6003
(9,1
0///08
002.0
012.3
655.000
05.2
C523
6902200
(902
C90222
0000
(339
[
CXL
1110
(B) Qwen2.5-Omni
Figure A4: Comparison of our audio-video frame interleaving with Qwen2.5-0mni's strategy.
single set of attention and FFN parameters to process both audio and video tokens. This poses a significant challenge when extending a pretrained T2V model with audio generation capability while preserving its original video generation performance, as validated in Tab, 2 To address this, JavisDiT (Liu et al., 2025c) introduces a dual-stream architecture with separate parameter sets for video and audio generation, along with ST-Prior and frame-level audio-video bidirectional cross- attention to enhance synchrony. However, this design introduces a large number of additional pa- rameters, making training 2025a), on the other hand, employs pretrained T2V model (Wan et al., 2025) and pretrained T2A model (Gong et al. 2025), with a complex cross-attention mechanism enable information exchange between audio and video, which still suffers from inefficiencies in both training and infer- ence stages.
difficult and inference
expensive. UniVerse-1 (Wang al.,
In contrast to these prior works, our proposed framework is simpler and more effective: it uses shared attention to enable cross-modal interaction between audio and video tokens, while adopting modality-specific FFNs to enhance intra-modal modeling quality. This design achieves a better trade-off between efficiency and performance, and provides higher scalability. This idea is also explored by concurrent work, Jo (Huang al. (025b), which further disentangles the QKV projectors for audio and video tokens within the self-attention block.
Design Difference with Qwen2.5-Omni's RoPE Strategy. Considering audio-video alignment, our TA- TA-RoPE design shares some similarity with Qwen2.5- 5-mni (Xu et al., (225)'ss RoPE solution: both approaches align audio and video through the position IDs along specific dimension to sure temporal alignment. However, as shown Fig. A4 there exists a key difference between us: whether the position IDs of audio and video tokens overlap. Qwen2.5-Omni treats audio, like lan-
21

---

Published as a conference paper at ICLR 2026
guage, as 1D token sequence and assigns the same position IDs across the three RoPE dimensions. This inevitably introduces overlaps between audio and video, e.g., an audio token with position ID 0,,0,0) 0) coinciding with video token (0, O, 0), potentially (25, 25, 25), (50, 50, 50), and so For multimodal understanding models such as Qwen2.5-Omni, such overlaps have negligible influ- non-trivial position confusion and lead to performance degradation, as quantitatively demonstrated in Sec, [.]
ence inference performance. However, for generative tasks like JAVG, overlaps
By comparison, our TA-RoPE design treats audio 2D image (represented by the form of mel- spectrogram (Liu et al. 2024b)). When ensuring temporal alignment by matching the 0-th dimension of position IDs between audio and video along the time axis, we introduce adaptive offsets in the other two dimensions of the mel-spectrogram corresponding to video width and height (as formu- lated in eq. 4 This strategy completely avoids overlap between the two modalities and enables more accurate audio-video synchrony.
bbuu Auni'' RoPE Strategy. Focusing on the domain of video-to-audio generation, AudioGen- Omiiii (Wang et al. (225c)) also identifies RoPE as a key component for achieving tem- poral alignment between audio and video, and proposes a specific design. However, due to the lack of detailed descriptions in their paper and the unavailability of their code, we are currently unable to conduct a thorough comparative analysis.
D ADDITIONAL EXPERIMENTS
D.1 ABLATION ON POSITIONAL. ENCODING STRATEGY
This section systematically investigates the impact of different audio position encoding strategies (illustrated in Fig. A5 on final audio-video generation quality, including:
Vanilla, ignores video positions entirely, encoding audio purely along the time and frequency axes of its mel-spectrooram: Ro (t, m) = (f,t, m))
Interpolate, aligns the audio temporal axis with video frames by interpolating intermediate Te.m. IDs: Ro(t, m) (t Tr T.
Interleave, aligns the audio temporal axis (0-th dimension) with the video temporal axis, while unfolding the other two dimensions along the mel-spectrogram: (a m) (1+ To f, m
7:
Interleave+Offset, similarly aligns the audio O-th dimension with the video temporal axis, but shifts the remaining two dimensions by the video width and height to fully avoid overlapping
position IDs between modalities: R. mt) = ([ To 1 + H, m w) (eq. (4)).
We conduct full audio-pretraining and AV-SFT training on 1- (Wan et al. 2025 to examine how these strategies affect both audio and video generation. Results Tab, A2 lead to three main conclusions:
1. Preserving integer audio position IDs is essential for audio quality -.eit Interpolate performs significantly worse, since frozen attention layers during audio pretraining cannot learn relative offsets represented by fractional IDs;
2. The more overlap exists between audio and video position IDs, the poorer the video quality becomes (Vanilla Interleave OnterleaveOOffset), consistent with our analysis in Sec. C
3. Generation models require non-overlapping position IDs to disentangle modalities. as well as temporal alignment along one axis for audio-video synchrony (e.g., Vanilla yields much worse synchrony than Interleave-based designs).
Based on these findings, we adopt Interleave+ as our final position encoding scheme.
D.2 INVESTIGATION ON TRAINING DATA QUALITY AND DIVERSITY
In this section, we conduct an in-depth investigation into the impact of diversity and quality of training data during the Audio- Video SFT stage. Specifically, we first construct low-quality dataset
22

---

Published as a conference paper at ICLR 2026
(000)
(00,1
(00,2)
(003))
(1,00)
(1000
0,0,
110,3
2400
(20.
222
240
Video
(0,1,)
01.
012
(0,33)
(1,1,0
011,1
0,1,2
111,3)
21,
0,1
2,1
121
(02,0)
(0,21))
(0222)
0,233
(1,20)
(122
1,22
0,233
220
22D
(22
(2233
(000))
(1,1,0)
(2200)
(3,,00
(44,0)
(3,00)
Audio
(Vanilla)
(0001))
(1,1,)
(22,1))
(3,3,1)
(4/1,1)
(5,5,5)
Audio
(000)
005000000
(1,1,0)
(1,5,115,0))
(2,2,0)
(25,22500))
(Interpolate)
(0,0,1)
(0550.511))
(1111)
(1.5,1551))
(2211)
(255,2511)
(0,00)
(0,1,0)
(1,2,0)
(1,3,0)
(2/40)
(2,5,0)
Audio
(Interlevve
(Q/011)
(01,1
(1,2,))
(1,3,1)
(241))
(2,5,1)
Audio
(0,3,4)
(044,)
(1,544)
(1,6/)
(27//9)
(2844)
(Interleave
+Offsat)
(0,35)
(04,5)
(1,5.5)
(1,6,))
(27,5))
(2,55)
Figure A5: Illustration of different audio positional encoding strategies.
Table A2: Ablation study on positional encoding strategy for both audio and video generation.
AndioCaps
JavisBench-mini (audio)
JavisRench-mini (video)
Strategy
JavisScore
DeSyne
AudioLDM2
5.06
0.200
0.460
8.81
0.153
0.360
DivisDiT-uudio
5.19
0.164
0.356
8.11
0.152
0.381
Vanilla
0.193
6.41
0.281
0.142
Interpolate
12.87
0.155
0.325
10.73
0.149
0.349
239.9
0.282
0.320
0.183
0.144
0.912
Interleave
0.144
Interleave+Oliset
4.65
0.198
0.420
6.81
0.154
0.417
221.3
0.283
0.317
0.200
0.153
0.807
of 720K text-aut audio-vvddee triplets by extracting large portion of speech videos (Gao et al. 2023
from the full data pool (see Sec. [B.2). Next, we follow OpenSora (Zheng et al. 2024) to filter out low-quality data using multiple scoring metrics such aesthetics (Schuhmann (022) (thresholding at 0.4), motion (Xu et al. 2023) (thresholding at 0.1), and OCR (Liao et al., (2020) (thresholding at 5.0), resulting in medium-quality dataset of 330K samples, which the adopted for AV-SFT in the main paper. Finally, we further raise the aesthetic threshold from 0.4 to 0.45, obtaining high-quality dataset of 100K samples for comparative analysis. After audio pretraining, conduct experiments with different data compositions in the AV-SFT stage, and the results are presented in Tab. A3 Accordingly, several conclusions can drawn:
First, data with high quality but low diversity cannot bring optimal performance. Models trained solely on the 120K high-quality dataset underperform those trained on the 330K medium-quality dataset. This is because transitioning from unimodal audio/video generation to joint generation is a relatively challenging task, requiring sufficient data quantity or diversity to enable the model to acquire new capabilities.
Second, data quality also indispensable. Models trained on the 720K low-quality dataset show clearly inferior generation quality compared to those trained the 330K medium-quality dataset. This degradation occurs because low-quality data undermine the priors learned by the Wan2. (Wan) et al. 2025) backbone during pretraining high-quality data. Even introducing second round of SET with 120K high-quality or 330K codium--qalityy data cannot fully recover the lost performance.
In summary, adopt the 330K medium-quaile dataset for AV-SFT training reasonable trade- off. We believe that further enhancing both the diversity and quality of training data will yield better scaling properties.
23

---

Published as a conference paper at ICLR 2026
Table A3: Ablation study on training data composition in the AV-SFT stage.
Quality
Consistency
Synchrony
Data
Quality
Epoch
FVD
FAD 4
TV-IB
TA-IB
AV-IB
JavisScore f DeSyne
120K
High
230.6
6.24
0,282
0.154
0.183
0.144
0.838
120K
High
2.0
239.9
5.81
0.283
0.157
0.189
0.151
0.811
120K
High
3,0
233.7
5.64
0.281
0.159
0.188
0.148
0.820
330K
Medium
1.0
225.5
5.62
0.283
0.161
0.190
0.145
0.822
330K
Medium
2.0
221.3
5.51
0.283
0.163
0.194
0.153
0.807
720K
Low
0.5
229.8
5.53
0.281
0.154
0.182
0.142
0.830
720K
Low
1.0
217.8
5.64
0.280
0.159
0.185
0.145
0.825
+120K
Low+High
2.0
223.8
5.46
0,282
0.161
0.191
0,150
0,823
+330K
Low+Med
212.0
5.45
0.283
0.160
0.187
0.146
0.824
D.3
ABLATION ON HYPER-PARAMERT
AV-DPO
In this section, further investigate the stability of hyperparameters in the AV-DPO algorithm proposed in eq. (6) focusing primarily on two factors: the choice 3 and the learning rate. Since different parameters influence the magnitude of the loss unevenly, we adopt implicit accuracy unified proxy metric to evaluate the impact of various hyperparameter settings:
n Acc E
I(Diffpalicy policy < Diff Acc
I(Diff
policy
< Diiff
(A1)
Recalling eq. (6), the implicit accuracy can measure whether the model successfully shifts toward the distribution the chosen data while moving away from that the rejected data. The experimental results, shown in Fig. A6land and Fig, A7 lead to the following conclusions:
implicit acc audio
implicit acc audio
0.55
0.85
0.50
0.75
0.75
9.70
heta,
0.50
0.50
bets, 1000
0.55
1o:5
beta, 3000
1r-50-6
0.55
hets 5000
0.50
1r le-6
0.4k
4,4k
8.4k
12.4k
16.4k k
20.4k
0.4k
4,4k
8.4k
12.4k
16.4k
20.4k
Seen Prompis
Seen Prompts
implicit acc video
implicit acc video
0.95
0.85
3.50
0.50
0.75
0.75
3.50
0.90
0.55
bets, 1000
0.55
1o.5
beta, 3000
Ir 5e-6
0.50
heta 5000
0.50
Ir 1e-6
0.4k
4.4k
8.4k
12.4k
16.4k
20.4k
0.4k
4.4k
8.4k
12.4k
16.4%
20.4k
Seen Prompiss
Seen Prompis
Figure A6: Implicit accuracy on 3 selections.
Figure A7: Implicit accuracy on learning rates.
Analysis 3 selection. In the original DPO algorithm, / primarily controls the divergence between the policy model and the reference model (Rafailov et al., 2023 Liu et al.) (025b). As shown in Fig. A6 audio achieves its best performance B 1000, whereas video converges faster and
24

---

Published as a conference paper at ICLR 2026
reaches higher final accuracy 3 3000. This observation is closely related to proposed framework. Our model built upon the pretrained Wan2.1-T2V, which already aligns well with human preferences; thus, larger 3 (e.g., 3000 or 5000) is needed to keep the video policy model closer to the reference model. contrast, the audio branch is newly trained and initially less aligned with human preferences, preferred data distribution. However, setting 3 too small (e.g., 100) leads to excessive divergence from the reference model and overfitting to imperfect preference data. Based these findings, set 3 3000 for the audio DPO loss and = 1000 for the video DPO loss to achieve relatively better performance.
smaller 3 (e.g., 1000) is required to shift the model closer to the
Analysis on learning rate. Fig. A7 presents three experimental groups with learning rates set x 10-5, 5 x 10-6, and x 10 respectively. The results show that 10 achieves both the fastest convergence and the highest final accuracy. This observation is consistent with prior works (Liu al., 2025b), which suggests that setting the learning rate in the DPO stage to approximately one-tenth of that in the SFT stage is suitable choice. Therefore, we adopt learning rate x 10-5 for training.
D.4 MORE VISUALIZATIONS
Fig. A8 and Fig. A9 showcase additional cases of joint audio-video generation, illustrating the strong generative capability of our proposed model across multiple dimensions.
25

---

Published as a conference paper at ICLR 2026
Prompt: small wooden cabin stands in a rainy forest, with misty trees around it and the sound of heavy rain and distant thunder.
Prompt: black-and-white image shows suited musicians stage playing saxophones, trumpets, and tuba, music filling the before curtain backdrop.
1000000000000000000000000 10 o io ( (DD
Prompt: man long curly hair beard plays electric guitar in studio, wearing black t-shirt and gray pants. Behind him are a monitor, speakers, and poster, as the sound of the guitar fills the room.
Prompt: group of sharks swim gracefully underwater, their fins and tails sending ripples through the clear water. The sound of gurgling and splashing follows their movement and the currents, with the sandy, rocky seabed visible below.
de ot ( (e 10 - (ww00oo
Prompt:A large cartoonish alien big eyes small mouth appears on screen, looking concerned. Behind dusk cityscape with tall buildings and a river sets the scene, as deep boom echoes, followed by softer one.
Figure A8: More examples for high-quality audio-video generation results.
26

---

Published as a conference paper at ICLR 2026
Prompt: woman red dress walks barefoot across sandy desert sunset, her hair flowing wind blows and faint footsteps sound in the sand.
Prompt: night, alley lined with traffic rope barrier, ground reflecting street lamps. Worn walls faded posters, and bright light glows at far end. sound of rain falls empty scene.
wat Mill-/oo (M end a feo0 (
Prompt: large waterfall cascades down rocky body below, creating dramatic scene. deep blue color, and there are greenish areas on the rocks near the waterfall. The sound of rushing water fills the air, steady and loud.
Prompt: young man with hair, wearing green shirt, plays the violin, halding under his drawing the bow produce clear, melodic tones. The background is plain wall, with soft shadows cast across it.
Prompt: yellow car with the number on it is driving on dirt road surrounded by trees, its engine revving up and then accelerating as the tires skid on the dirt road.
Figure A9: More examples for high-quality audio-video generation results.
27


Reviews:
[{"review_id": "1jtFXvmYom", "reviewer": "Reviewer_9Rrw", "summary": "This paper proposes a new method for joint audio-video generation. In contrast to most prior works that model the two modalities as two separate streams, this paper proposes to use a mixture-of-experts design to allow frequent communication and joint modeling of both modalities. This paper also proposes to be a temporally aligned rope encoding scheme and a DPO method for preference alignment. The proposed approach achieves state-of-the-art performance.", "questions": "- Will the code and model be open-sourced?\n- Will the filtered training dataset be released? This would be a significant contribution.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The joint modeling of audio and video tokens with separate FFN layers is simple and elegant, although these ideas already exist in prior works (e.g., BAGEL) under different contexts.\n- The presented model has a strong empirical performance, setting a new state-of-the-art for the challenging problem of joint audio-video generation.\n- The study of different finetuning strategies (e.g., Lora with different ranks) is interesting. It provides additional context and insights to the readers.\n- The ablation study on data composition in the appendix is also informative.", "weaknesses": "- The proposed network seems to be significantly larger than the networks used in prior works. Can the authors also include the parameter counts and runtime in Table 1 for a more comprehensive comparison? \n- The gains in Table 4 from using DPO seem to be very limited, except in FVD. How important is DPO? Are there any user studies or qualitative examples that support the use of DPO? \n- There are no details on training data filtering. Section D2 contains a high-level sketch, but it does not mention which algorithms or what thresholds were used.\n- The last update date of the linked repo is after the ICLR deadline.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6SfFRmA5rJ", "reviewer": "Reviewer_qjFx", "summary": "This paper proposes a joint audio-video generation (JAVG) model built on the pretrained text-to-video model Wan2.1.\nTo enable synchronized audio and video generation, three mechanisms are introduced: a model design called MS-MOE for joint generation, a new RoPE variant (TA-RoPE) that manages both cross-modal interaction and single-modal structure, and an audio-video direct preference optimization method (AV-DPO) that aligns the model with human preference. \nExperimental results demonstrate that the proposed method outperforms existing baselines.", "questions": "- Did the authors evaluate the model performance after the Audio-Video SFT stage, prior to AV-DPO finetuning?\n- For TA-RoPE, could the authors visualize or analyze the similarity structure between RoPE embeddings used for audio and video?\n- For the training data used in AV-DPO, how did the authors sort the videos based on three scores?\n- Is AV-DPO applicable to other existing JAVG models?\n- Did the authors conduct a user study to compare the proposed method with baselines?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed model design (MS-MOE with LoRA finetuning) is simple yet effective, achieving both a high cross-modal alignment and a high single-modal generation quality. It is also computationally efficient, as inference cost per token remains constant while the parameter size increases.\n- The proposed TA-RoPE requires minimal engineering efforts. It provides a natural extension of Wan's RoPE to support both inter- and intra-modal interactions of audio and video.\n- AV-DPO sounds novel, as applying DPO for aligning JAVG models with human preference is underexplored.", "weaknesses": "The primary concern lies in the **fairness and clarity of the experimental evaluation**.\n\nFor Table 1:\n- It is unclear which T2A models are used for T2A+A2V and which T2V models are used for T2A+V2A. Also, many results of T2A+A2V and T2V+V2A baselines are missing. \n- The authors re-evaluate JavisDiT, but the reported scores deviate substantially from those in the original paper. For instance, TA-IB = 0.197 (original) vs. 0.151 (this paper), CLIP = 0.325 vs. 0.308, AV-IB = 0.201 vs. 0.197, AVHScore =  0.183 vs. 0.179, and JavisScore = 0.158 vs. 0.154. These metrics consistently decrease, and in some cases, the original scores are higher than those of the proposed method.\n- Only the final model (finetuned using AV-DPO) is reported. It would also be important to show results after the Audio-Video SFT stage to isolate the contributions of the model architecture and AV-DPO optimization.\n\nFor Fig.2:\n- All evaluation metrics appear identical to those used in AV-DPO training, relying on the same reward models. Since only the proposed model is optimized for these metrics, this evaluation is biased and not directly comparable to other baselines.\n\nFor Fig.7:\n- The definitions of A-LoRA, A-noLoRA, AV-AttnLoRA, and AV-LoRA are missing, making the figure difficult to interpret. A more detailed explanation is needed.\n\n\n**Lack of justification for the TA-RoPE design.**  \nWhile the reviewer agrees that TA-RoPE is a reasonable engineering extension, the motivation behind the specific positional correspondence between audio and video modalities is not fully explained.\nIn particular, the mapping of video height and audio time, and video width and audio frequency, may introduce implicit correlations that lack perceptual grounding.\nA brief visualization or analysis of the positional-similarity structure would help clarify how the TA-RoPE affects the interaction between audio and video modalities.\n\n\n**Lack of subjective evaluation.**  \nAlthough the paper claims that AV-DPO improves perceptual alignment with human preference, no user study is conducted to support this claim.\nIncluding even a small-scale user study would make the evaluation more convincing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pg9ImQ0Urf", "reviewer": "Reviewer_uNjt", "summary": "This paper presents a new method for efficiently constructing a model capable of jointly generating audio and video. Given a pretrained DiT-based video generation model, the proposed method first adds audio-specific FFNs to each DiT block, and these added FFNs are trained with text-audio pairs for text-to-audio generation. Then, the entire model is fine-tuned in a parameter-efficient manner using text-audio-video triplets for joint audio-video generation. Finally, the model is further fine-tuned by direct preference optimization with a dedicated audio-video preference dataset. Experimental results show that the model obtained through the proposed method achieves superior performance compared to existing models.", "questions": "- Why are several numbers (especially DeSync in sequential generation baselines) missing from Table 1? These are necessary for a comprehensive comparison of joint generation performance.\n- What models are used for T2A (or T2V) in the sequential generation baselines?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The proposed method introduces a novel approach for efficiently constructing joint audio-video generation models. While many prior works use a separate audio branch in addition to a video branch, the proposed model only adds dedicated FFNs for audio.\n- The proposed positional encoding is carefully designed to leverage pretrained video generation models while inducing strong audio-visual alignment via temporal encoding.\n- This work explores direct preference optimization for joint audio-visual generation. The preference data is automatically constructed using several reward models. This approach is simple and reasonable, and could serve as a good baseline for future research.", "weaknesses": "- While the idea of applying DPO to joint generation models is interesting, its effect appears to be relatively small compared to the other proposed modifications. It would be beneficial to investigate the potential causes of this result. \n  - For example, if the quality of the preference data is a factor, this could be clarified by a subjective evaluation of a small subset of the preference data. If the amount of preference data is insufficient, showing performance as a function of the amount of preference data used would be helpful.\n- Some visualizations could be further improved:\n  - Showing spectrograms rather than waveforms would be more informative for illustrating the semantics of the audio content.\n  - What does the vertical axis in Figure 7 represent?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BcKktp4xQx", "reviewer": "Reviewer_Zh7Q", "summary": "The paper presents a new framework for audio-video joint generation, which features three new techniques. First, a modality-specific mixture-of-experts (MS-MoE) design is adopted, which leads to both cross-modal alignment and single-modality generation quality. Second, a temporal-aligned RoPE (TA-RoPE) strategy is introduced for facilitating frame-level synchronization between audio and video tokens. Third, an audio-video direct preference optimization (AV-DPO) method is applied to align model outputs with human preference across quality, consistency, and synchrony dimensions. Experimental results with a model built on Wan2.1-1.3B demonstrate that the proposed framework outperforms previous approaches both qualitatively and quantitatively. Ablation studies justify the design of the proposed modules.", "questions": "I have a question/concern, which I provided in \"Weaknesses\". I would appreciate it if the authors could share their thoughts on it. If the authors' response is convincing, I will raise my rating.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Although there is room for improvement in presentation, the paper itself is written well enough to make readers understand the authors' motivation, the proposed method, and the experimental results.\n2. The proposed framework is well-designed. The ablation studies are comprehensive and support the design.\n3. The experimental results are good both quantitatively and qualitatively.", "weaknesses": "Although the proposed framework is well-designed and works better than the previous ones (JavisDiT and UniVerse-1), there is one concern/question. Newly introduced FFNs for audio tokens will modulate audio tokens so that the attention layers trained for video tokens can deal with audio tokens as well. This idea is great. However, as for the first Transformer block, audio tokens (the output of the audio encoder) will be fed into the self-attention and cross-attention layers as they are. It is suspected that the self-attention and cross-attention layers in the first Transformer block will not be able to deal with audio tokens very well. So, inserting another FFN between the audio encoder and the first Transformer block and training it in the pretraining phase might help the first self-attention and cross-attention layers process audio tokens.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)