JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation
Authors: Kai Liu
Jungang Li
Yuchong Sun
Shengqiong Wu
jianzhang gao
Daoan Zhang
Wei Zhang
Sheng Jin
Sicheng Yu
Geng Zhan
Jiayi Ji
Fan Zhou
Liang Zheng
Shuicheng YAN
Hao Fei
Tat-Seng Chua
Venue: NeurIPS 2025 spotlight ()
Keywords: Multi-Modality
Large Language Model
Souding-Video Comprehension and Generation
Primary area: deep_learning

Abstract: This paper presents JavisGPT, the first unified multimodal large language model (MLLM) for joint audio-video (JAV) comprehension and generation. JavisGPT has a concise encoder-LLM-decoder architecture, which has a SyncFusion module for spatio-temporal audio-video fusion and synchrony-aware learnable queries to bridge a pretrained JAV-DiT generator. This design enables temporally coherent video-audio understanding and generation from multimodal instructions. We design an effective three-stage training pipeline consisting of multimodal pretraining, audio-video fine-tuning, and large-scale instruction-tuning, to progressively build multimodal comprehension and generation from existing vision-language models. For instruction tuning, we construct JavisInst-Omni, a high-quality instruction dataset with over 200K GPT-4o-curated audio-video-text dialogues that cover diverse and multi-level comprehension and generation scenarios. On JAV comprehension and generation benchmarks, our experiments show that JavisGPT outperforms existing MLLMs, particularly in complex and temporally synchronized settings.

JavisGPT: A Unified Multi-modal LLM for Sounding-Videe Comprehension and Generation
Kai Li Jungang Li Yuchong Sun Shengqiong Wu², Jianzhang Gao Daoan Zhang". Wei Zhang". Sheng Jin?, Sicheng Yu", Geng Zhan", Jiayi Ji², Fan Zhou', Liang Zheng"", Shuicheng Yan", Hao Fei Tat-Seng Chua?
'ZJU, "NUS, SHKUST(GZ), 'RUC, SUR, "HZCU, 'NTU, SMU, 9USYD, 10 ANU
Project: hutps:://avieeVeree.ggii wtw..i....o....ii/aattaaa
Entity Localization
Instruction- F Followed Generation
Proactive Conversation
pou8l lkmilid JaS
May? mani playgg Sir galbee
RN pansud non
Pwo sad veno LELOUTE PARKy miceabiic Sacng
Fiokked aneadig
Jee ofr Chon?
stiiito aader 22 whewe loaal Lotole
Relationship Comprehension
Tha weads like nenext An Anper prides sutterr 21, vocalit
In-contest Reasoning
Icrdd whirti the eve dight guoiing
Iike't viko, wbote souumad. Oone
Tie lodo grout the you comon the volous
Global Perception
Somp odburrad
Aboleseby (Tiepped for vinu gigiigiing
Hist the starppiie the wide and
tusher the E3INC
pnoikeeraennn wring eallbiing rervc ty
Figure I: JavisGPT supports multi-level synchronized audio-video content (i.e., sounding video) comprehension (left), and simultaneously complex instruction-basee audio-video generation, inter- leaved in-context generation (middle), and multi-turn proactive conversations (right).
Abstract
This paper presents JavisGPT, the first unified multimodal large language model (MLLM) for joint audio-video (JAV) comprehension and generation. JavisGPT has concise encoder- L.M--deoodrr architecture, which has a SyncFusion module for spatio-temporal audio-video fusion and synchrony-awan learnable queries to bridge pretrained JAV-DiT generator. This design enables temporally coherent vidco-audio understanding and generation from multimodal instructions. We design an effective three-stage training pipeline consisting of multimodal pretraining, audio-video fine-tuning, and large-scale instruction-tuniin to progressively build multimodal comprehension and generation from existing vision-language models. For instruc- tion tuning, construct Javis -0mii high-quality instruction dataset with over 200K GPT-4o-curated audio-vidde video-text dialogues that cover diverse and multi-level comprehension and generation scenarios. On JAV comprehension and generation benchmarks, our experiments show that JavisGPT outperforms existing MI.LMs, particularly in complex and temporally synchronized settings.
'Equal contribution. Work done during Kai Liu's visiting period at NUS. Email: kail failzzja aailezju.edu.cn Corresponding author. Email: haofei7419@ gmail.com
39th Conference on Neural Information Processing Systems (NeurIPS 2025).

---

JavisGPT
This is the adjusted video with closer finger moverenns (audio uncbanged).
JAV-DiT
Does work for you?
MLP
Lora
Backbone Large Language Model (Qwen2.55VLL-7 5-11-77--IIntrreee
Learnable
SyncFusion
This video perspective bad. So many details finger movements are lost. Text Prompt
Merger
MLP
SyncAV Token
Text (in/out) Token
Visual Encoder
Audio Encoder
JavisQuery Token
JavisCond Token
Figure 2: Overall architecture of JavisGPT, which can perceive and produce videos and sounds simultaneously. The input video and audio are encoded and fed into the SyncFusion module, and the resulting synchronized audio-video representation, together with the text tokens and learnable JavisQueries then passed the LLM backbone. During decoding, the yielded Javi sCond em- beddings are used to align the LLM intents to the semantic conditional space of downstream JAV-DiT; enabling high-quality and synchronized sounding- generation.
Introduction
Joint understanding and generation of multiple modalities has recently sparked increasing interest [7] 80] 21 86 (5). Existing efforts generally focus on image-text frameworks [96] 84 73 11
or treat video and audio understanding and generation as separate modalities [90] 81 42
We introduce JavisGPT, a bespoke MLLM for high-quality, unified comprehension and generation in joint audio-video (JAV) tasks, which has clear application potential in avatar chatbots, sounding movie curation, and music-driven video analysis. JavisGPT a holistic system that accepts as input separate audio and vidcos, synchronized sounding vidcos. and user text prompts. After comprehending and reasoning over these input signals, it produces either textual responses and/or synchronized sounding videos. Our architecture supports wide range of JAV scenarios, including question-answering in comprehending sounding audios, generation of sounding audios following user instructions, upon in-context reasoning, and interleaved text-audio-vide generation in multi-turn user dialogues (ref. Fig. IT. As a comprehensive and state-of state-of-the-ar system, we highlight the design of JavisGPT below (with detailed discussions with related works provided in Sec. 2).
Model architecture. JavisGPT uses an encoder-L.L. M.tiee--LLMMMeeooeee architecture (ref. Fig. 2p. with Qwen2.5 (87 as the LLM backbone. The visual encoder is inherited from Qwen2.5-VL (5]. and the audio encoder is based on BEATs [10]. Audio and video features, along with user prompts and Icarnable JavisQuery tokens, passed to the LLM. To enable fine-grained spatiotemporal alignment, we propose a dedicated SyncFusion module that fuses audio and video representations into synchronized SyncAV tokens for unified comprehension. At the output stage, the LLM generates textual responses along with JavisCond tokens, which encode contextual semantics and serve as conditioning inputs for pretrained JAV-DIT generator (39]. We choose JavisDiT for its generation quality and flexibility, and incorporate hierarchical JavisQueries to provide spatiotemporal priors, further enhancing synchronization in audio-video generation.
Training strategies. To effectively adapt the languaeee backbone to JAV tasks, we design a three-stage training pipeline: (1) MM-PreTraine introducing the audio input branch for comprchension and preliminarily aligning the output embeddings of LLM with the condition space of JavisDiT: (2) AV- FineTune; enhancing synchronized audio-video comprehension and generation via SyncFusion and hierarchical JavisQueries, respectively; and (3) IMM-nntTumm:: enables generalizable instruction following and multimodal reasoning via large-scale instruction tuning across text, video, and audio.
Dataset collection. To facilitate instruction tuning, we construct IavisInst-Onn' a large-scale, diverse, and high-quality instruction dataset covering both comprehension and generation. The dataset contains 200K dialogue trajectories involving tightly interleaved text, audio, and video, simulating complex single- and multi-turn interactions. All samples are annotated via GPT-4o and manually

---

verified for quality. The instructions span wide range of task formats, including QA, captioning, connuction-folooiinn generation, and in-context multimodal reasoning.
We compare JavisGPT with existing JAV-like LLMs on various JAV benchmarks, where JavisGPT yields impressive performance across all scenarios. Moreover, for audio-video joint generation, JavisGPT also significantly outperforms existing methods, producing high-quality and diversified sounding videos. We reiterate our main points below:
We present JavisGPT, the first unified MLLM for sounding-vide videoo comprehension and generation. We present its architecture design, including the audio-video synchronization mechanism, its multi-stage tuning strategy, and a large-scale instruction dataset for JAV LLMs training. Evaluation indicates the effectiivness of JavisGPT in JAV comprchension and generation.
2 Related Work
Multimodal comprehension. Recently, MLLMs have significantly advanced many multimodal
fields, such as the modeling of image. video, and audio contents [5] 12 18
60
22
20 79 (55)
Early approaches, such as Flamingo [4] and BLIP 33 34, focused on combining text with static visual inputs. Recent research extended these efforts to dynamic vision and audio modalities, such as PandaGPT [67]. Video-LLaMA 14, Unified.MILLM (36.. and MiniOmni [85], which incorporate temporal and auditory perception and reasoning. While these MILMS achieve promising performance, they typically treat modalities (e.g., audio and video) as independent inputs from separate channels, relying on concatenation simple feature padding for integration. This strategy overlooks fine- grained spatio-temporal interactions between modalities, compromising their effectiveness in complex scenarios where multimodal synchronization is required, such as JAV.
Multimodal generation. Compared to research efforts for multimodal content comprehension, relatively limited emphasis has been placed generative capabilities for MLLMs, particularly for scenarios involving multiple interdependent modalities, such as JAV [31 55 For audio-video generation, existing MILMs like HuggingGPT (64, NEXT-GPT [8), and UnifiedlO-2
442 adopt
pipeline-based framework, where individual audio and video generation components are sequentially executed. Although such methods achieve reasonable results, they suffer from issues like error propagation and desynchronized outputs due to the lack of joint modeling.
Unified comprehension and generation. Early approaches such as AV2AV 15 and MultiDia- log 52 tackled talking-hear head generation and speech-driven dialogue, but were limited by simple architectures or the absence of textual input/output. While recent MLLMs aim to unify multimodal comprehension and generation, most still focus on single modalities, lacking effective modeling of cross-modal interactions, particularly the audio-video synchronization. For instance, Video- Salmonn 688 70 and Meerkat [6] use separate encoders for audio and video with little attention to alignment. Similarly, NEXT-GPT [81] and AnyGPT [90] support multimodal generation but fail to synchronize audio and video due to separate decoders. To address these limitations, we propose JavisGPT, which integrates a spatio-temporal alignment mechanism and unified feature learning, advancing the state of the art in joint audio-video (JAV) modeling.
3 Architecture
JavisGPT leverages Dween.5-vVL-7B-Insteee [5] backbone, incorporating with SyncFusion and JavisQueries to perform joint comprehension and generation. Sec. [.] demonstrates how Javis- GPT extends audio perception and utilizes SyncFusion to explicitly capture spatio-temporal synchrony in input sounding videos. Sec. 3.2 introduces how JavisGPT integrates with a pretrained JAV generator 39 and enhance the generation synchrony through hierarchical JavisQueries.
3.1 Synchrony-aware Audio-Video Comprehension
70,, we first adopt the vision encoder from Qwen2.5-VL [5] to encode the video inputs to yield dense video representation V (HxW)XC where T, denotes the number of video frames, (H, W) are the spatial dimensions, and C the feature dimensionality. Meanwhile, we introduce a parallel audio encoding branch built upon BEATs [10) to extract dense audio representations A R XXC
modality-speciinn encoding for video and audio. Following recent works 14 68

---

Frame
Frame
Frame k
Audio Token
Visual Token
@ 3
@ +
Shared df...
GreesAttn
Shared-R...
5666 6bbb
SyncAV Token
Figure 3: Mechanism of Synneuusion. (1) Left: We align camporall-ssemeent audio tokens with corresponding video frames by using cross-attention to merge audio clues into visual patches, so as to capture spatiotemporal synchrony explicitly. (2) Right: Each resulting SyncAV token et E R represents a sounding event occurring within the i-th row, j-th column visual patch of t-th frame.
where Ta is the temporal length of the audio features, M is the number of frequency bins, and C is the feature dimension. Specifically, the raw audio waveform first converted into a 2D mel-spectrogram, comprising T. time frames and M = 128 frequency bins. Each audio frame corresponds a 25 ms window with a 10 ms overlap, enabling high-resolution temporal encoding, Finally, two-layer MLP og is applied to project audio features into the LLM token embedding space, serving as a bridge for JavisGPT to understand audio content.
Synchronized audio-video fusion. Unlike previous MLLMs that straightforwardly concatenate video and audio features together [14, 68) or simply interleave audio-vides video frames [70, 86], Jav- isGPT introduces a SyncFusion module v/av to explicitly model the spatio-temporal synchrony in sounding videos. As illustrated in Fig. 3 we first temporally align the audio and video streams by evenly dividing the audio representations to match the number of video frames, transforming the audio [Ta/ /T. with sero-paddings
features from A E RTxxMxC to A' E T M)xC where applied if needed. Then, a shared cross-attention module is employed to inject audio cues into the frame-wist wise video representations, followed by two-layer MLP generate SyncAV representations. The length of SyncAV representations matches that of the vanilla extracted video representations, ensuring compatibility with the MRoPE positional encoding used in the Qwen2.5-VI. (5). Each re- sulting token e.,, E RDC encodes a localized audio-visual event occurring at the i-th row, j-th column visual patch of frame by. By integrating audio semantics into visual features, the SyncFusion module enhances representational expressiveness and achieves precise spatio-temporal synchronization. This crucial for downstream tasks requiring fine-grained audio-video understanding.
3.2 Spatio-Temporally Synchronized Audio Video Generation
Equipping backbone MILM with DiT generator. Drawing inspiration from Pan et al. [51. we introduce N learnable query embeddings Q' to bridge the hidden state space of the LLM with the semantic condition space of the DiT generator. As demonstrated in Fig. when JavisGPT encounters the special token < sva__stt aa__tart| | during the NTP process, the N learnable tokens are immediately inserted into the input sequence. These tokens are processed through the causal attention layers of the I.M,, after which two-layer MIP &b are applied to project the N' hidden states h TR NxC into the condition embedding space of the DIT, yielding é R XDD, which serve semantic conditions for generating synchronized sounding videos. Unlike MetaQueries [51], which completely replaces the original conditions with è and require full fine-tur fine-tuning of the DiT, the proposed JavisGPT retains the original DiT parameters by freezing the generator and instead aligns è with the original TX-XXL [59] text encoder with alignment loss Latign =II é ||a [8I). This design greatly simplifies training and reduces computational cost while maintaining effective conditioning for generation, and Sec.
provides further discussions with other generative MLLMs.
Synchrony-enhanced audio-video generation. To further strengthen the spatio-temporal synchrony of generated sounding videos, we follow Liu al. [39] to additionally design N' learnable query tokens Q' to estimate spatio-temporal prior (ST-prior) embeddings $ for auxiliary conditions. As shown in Fig. the semantic condition a coarse description of the overall audio-visual event (e.g., 'a car starts its engine and leaves the screen'), ), while the ST-prior condition á captures fine-grained synchrony. Specifically,, the spatial prior governs where events occur and disappear (e.g., 'the car in the tep-left corner of screen'), ) whereas the temporal prior determines when they begin and end

---

Text
Hierarchical JavisCond
Text/Prion Encoder
GT JAV-Caption
Response
Alignment Loss
Loss
MLP
Training Inference Unused
Semontic Condition
ST-Prior
Condition
Backbone LLM
JAV-DiT
Noise
Multimodal Prompt
Hierarchical
JavisQuery
Diffusion Loss
Figure 4: Instruction-follonn followed and synchronized audio-video generation. We use learnable queries to gather useful information from all the audio, video, and text inputs, and use a two-layer MLP to map the hidden states from LLM the conditional space of DiT. The hierarchical semantic and spatiotemporal prior conditions further enhance the synchrony in generated sounding videos. Alignment loss and diffusion loss are integrated to reduce optimization difficulty and training cost.
(e.g., 'sound starts 2s, exits 7s, fades 9s'). To compute 8, employ separate MLP projector 66 to map the final-layer representations of the prior query tokens into the ST-prior condition space, aligning them with the ST-prior encoder used JavisDiT [39]. Finally, we combine the alignment loss between the predicted ST-prior embeddings é and the ground-trun ST-priors s with the denoising objective, jointly optimizing both strengthen the audio-video synchrony:
Leomp =I| - c ||2 + |
||2 +
|| E(ar, Vt, č, 1) ||2 - Lasson Calign +
LC.S
diff
(1)
where é(*) is the DiT model that predicts the noise/velocity added to video v and audio a at timestamp t with (estimated) condition and st-prior condition and is the ground- We follow Liu
et al. 39 to use rectified flow 440 as the scheduler.
4 Training and Datasets
This section presents the three-stage training pipeline designed to efficiently adapt the visual-language backbone MILM [5) into our proposed JavisGPT for joint comprehension and generation of sounding videos. A brief summary of the training process is demonstrated in Tab. with detailed training configurations provided in Sec. B.2
Table I: Summary of the training pipeline. 'MM-PT" denotes multimodal pre-training, 'AV-FT"
refers to audio-video fine-tuning, and 'MM-Ins(Tune' means multimodal instruction-luning.
Stage
Tasks
Trainable Modules
Parameters
Training Objective
MM-PT
A-T, T-AVV
Q de.s th Qur s yo Qui s. LLM
239.95M
Lutp + Latign Cntp + Latign + Laist
AV-FT
AV-T, T-
654.68M
MM-InstTune
A/V/AV-T-IT+V
717.09M
Latp + Lutign + Lasst
4.1
Pre-training for Basic Video-Audio Comprehension and Generation
The first stage equips JavisGPT with basic AV comprehension and generation skills by extending Qwen2.5-Vi VL to audio and aligning it with the JavisDiT generator.
Audio input alignment learning. As described in Sec. 3.1 we optimize the only parameters of the two-layer MLPs Φ", which are applied to project the audio features (encoded by BEATs [TO] to the input token embedding space of LLM. Specifically, following the setup of VideoLLaMA2 [14, train 6t on approximately 600K audio-text pairs using audio captioning and audio- based question- answering tasks by minimizing the next-token prediction loss Cup the generated text responses.
Preliminary audio-video generation alignment learning. In this part, we learn learnable query embeddings Q' along with the projection network &e to map the L.M''s hidden states into the condition space of the DiT generator, which is originally modeled by TS-XXI. encoder [59]. Specifically, The learnable query embeddings Q aggregated contextual information from the LLM
5

---

Calegad Walleer
Algreen EX
Lotte-i deo
TANERESKY Set
Areoshher
56
UnUnd
A Und
Grturdin
Lecaliastion 35K
Global 25K
Sm
Malt- Turn-
AV-
AV.D
Courreg
In-Context 30k
AV-Lnnd
Jentinst
Reintlonddip 50K
30
1244
TAvisbechhaa Sop
Spand
ADD
LiKK
IBc
Details
JavisInst-Omni
JavisInst-Und
JavisInst-Gen
Figure 5: (1) Left: We curate a large-scale, diverse, and balanced cross-modal instruction- tuning dataset (JavisInst-Omni) from multiple sources. (2) Right: Within JavisInst- Ommi., JavisInst Und and JavisInst-Ger are specifically developed for multi-level audio-video com- prehension and generation with synchrony-awarene Details are presented in Sec.
Specifically, we use 1.5M sounding video captions from TAVGBench [46] as input prompts, each concatenated with the N queries, which serve aggregate contextual information from the LLM. Then, the resulting hidden states Q corresponding to these queries are projected by o to obtain the estimated condition embeddings č. To accelerate and stabilize the semantic alignment between the LLM and DiT, we minimize an alignment loss Latign between the predicted conditions è and the reference conditions encoded by the frozen T5-XXL from the captions. This pre are-alignment phase is crucial for providing semantically meaningful initialization, which in turn facilitates subsequent training of fine-grained audio-video synchronization.
4.2 Fine-tuning for Synchronized Audio-Video Comprehension and Generation
Further, the model is fine-tuned to enhance its ability to capture fine-grained audio-video synchrony. 128, 256) is integrated with the LLM backhone strengthen the adaptation.
LoRA
27
Synchrony-aware audio-video comprehension. This part focuses on optimizing the SyncFu- sion module visv to strengthen spatio-temporal alignment of the audio-video representations. The module trained on the sounding-vinn caption task, where sounding videos are provided as in- put, and the LLM is required generate visual and acoustic descriptions. We use around 360K andio-video-terr triplets from TAVGBench [46], with supervision provided by the NTP loss.
Synchrony-awand awwre audio-video generation. In this stage, we further fine-tune the semantic and st-prior queries Q", Q* along with the output projectors o² d², aiming to improve the synchrony of generated sounding videos. Training is conducted by minimizing the loss defined in Equation ( allowing the model to better align semantic intent with fine mee-grained spatio-temporal dynamics during generation. We reuse the collected 360K audiovvideo-tex triplets from TAVGBench for this stage.
4.3 Instruction-tuning for Interleaved Comprehension and Generation
As an interactive chatbot, JavisGPT is expected to not only equipped with instruction -osllotton-follooinng ca- pabilities but also improve its ability reason over user intent for more generalized and user-friendly interaction. To this end, we first construct a large-scale, diverse instruction tuning dataset, named JavisInst-Omnt to enable flexible and interleaved audio-video comprehension and generation. The overall dataset statistics are depicted in Fig.
JavisInst- Omnni dataset. First, to maintain model capacity of comprehending single video and audio modalities. we sample diverse QA instances from existing resources: 55K audio QA samples from Lid2 [14, 60K video QA samples from LLaVA-Video- 178K [95]. and 20K image QA instances from for 95K audio-video QA instances from VideoLLaMA2 [44 and 20K audio-video caption samples from TAVGBench [46), we additionally construct
[32] Then, to enhance joint audio-video comprehension, except
for multi-level synchrony-awan comprehension. Finally, to strengthen synchronized sounding video generation, except for 150K text-to-audio-vide instances from TAVGBench [6 to maintain JAVG capability, we further curate JavisInst Gen to handle various conversation scenarios. Sec. provides more details.

---

Table 2: Comparison on audio, video, and audio-video comprehension. We the ActivityNet [6] Perception [53. and MVbench [35] test sets for video comprehension, the ClothoAQA (38 and TUT2017 49 test sets for audio comprehension, and the AVQA [8), MU-AVQA [1], and AVSD
test sets for synchronized audio-video understanding. JavisGPT exhibits competitive performance
across all benchmarks. We also report the data (#Samples) used adapt backhone VLM for joint
audio-video comprehension. Best and secondary results marked bold and underline respectively.
Video
Audio
Audio Video
(7-8B)
#Samples
Aetivity.et
Perception
MVbench
ClotheQQ
TUT2017
AVQA
MU-AVQA
AVSD
Video-LLLM
Video-LLa.V
27
56.5
67.9
58.6
MaVA-NeXT
46.5
LLaVA-OV
32
55.6
57.1
56.7
Qwen2-VL
76
57.4
62.3
67.0
Qwen2.5.V.
70.5
IntemV11.255
58.9
68.9
72.0
Andio-LLM
Owen-Audi
17
57.9
64.9
Qudio-Audio
18
60.9
73.7
NEXT-GPT [1
1.9M
21.5
33.7
27.9
30,9
6.4
25.3
19.3
30.8
Unitidd0-2
9.2B
23.2
34.7
30.4
31.4
9.1
61.2
34.1
16.5
Macaw-LLMh LLM
78.7
31.8
34.3
AV-LLM
65
47.2
78.7
45.2
$2.6
Lad2
[4]
1.9M
50.2
51.4
54.6
65.1
78.4
79.2
57.2
LMMA2
77.3
Qwen2.5-Omni
80
57.2
70.4
66.9
68.0
78.3
91.5
79.9
62.8
JavisGPT (Ours)
1.5M
58.1
70.2
68.4
67.3
82.1
93.8
82.1
62.2
Joint instruction-tuni -tuning across modalities. To enable JavisGPT supports diverse tasks shown in Fig. we jointly fine-tune all newly introduced modules, including input/output projectors d" th 6", o", learnable queries Q", Q", and the LoRA adapter IlU on diversified cross-modal instruction-tuie tutiinn instances from our proposed JavisInst -mmn dataset.
Contribution Statement
This work is the result of large and comprehensive project that aims to provide the community with an open-source tool and platform for joint understanding and generation sounding video, which is the first of its kind. While most of the individual network components are known in the literature, it takes significant trials and errors to find a feasible plan, and connect and align them to form state-of-the-an and yet concise system. In particular, the proposed three-stage training pipeline has proven useful for gradually improving model performance. In addition, the proposed Javis Inst-Ommt Omni dataset represents very rich mixture of single- and multi-modality data, with the majority being single-turn and the rest multi-turn. We believe the architecture, training methods, and datasets offer very competitive baseline and have significant value for future data-centric and model-centric studies.
Experiments
This section presents the experimental results to evaluate the effectiveness our proposed JavisGPT. For fair comparison, we train JavisGPT following the three-stage pipeline described in Sec. 4 with each stage running for one epoch. We use zero-shot evaluation for downstream tasks including audio-video comprehension and generation using their official evaluation protocols. Additional implementation and training details can be found in Sec. B
6.1 Comparison with the State of the Art
Multimodal comprehension. As shown in Tab. our method achieves performance on par with advanced models on video/audio understanding, demonstrating strong unimodal capabilities. It main- tains visual comprehension strength of Qwen2.5-VL (5) backbone (70.2 vs. 70.5 on Perception (53)

---

Table 3: Comparison on audio-video generation. Results are reported on JavisBench mini [39
We achieve stronger generation performance than standalone DiT models and unified MLLMs.
AV-Quality
AV-Consistency
AV-Synchrony
Method
FVD!
KVDJ
FADI
TV.I8T
TA-IB1
CLIP?
CLAPT
AV-IBT
CAVPT
AVTIScoref
JavisScure
DIT
MM-Diff (0)
2311.9
12.2
27.5
0.080
0.014
0.181
0.079
0.119
0.783
0.109
0.070
JavisDiT
39
248.9
1.9
7.6
0.273
0.145
0.307
0.382
0.203
0.799
0.181
0.153
MILM
NEXT-GPT
81
1463.2
6.1
7.0
0.215
0.132
0.182
0.371
0.067
0.732
0.061
0.038
UniftddOO22
42
1597.4
6.4
6.7
0.231
0.118
0.292
0.357
0.106
0.747
0.094
0.053
JavisGPT (Ours)
2.6
0.146
0.309
0.380
0.202
0.157
and even slightly outperforms Qwen2.5-Ommi [66] MVBench 335
(video) and TUT2017 49
(audio). For audio-video synchronized understanding, JavisGPT surpasses both Owen2.55-mmm and Vide2LLaMA2.1 [1] on AVQA [89] and MU-AVQA [41], achieving top results on both benchmarks. We attribute this to the spatio-temporal alignment capability of the proposed SyncFusion and our synchrony-avee aware training strategy. Notably, those results are obtained with fewer training samples (1.5M in total), highlighting its strong data efficiency.
Multi-modal generation. Tab. 3 compares the tit-oo-audio-vide generation performance of JavisGPT with existing methods. JavisGPT significantly outperforms NEXT-GPT 81 and UnifiedIO- 2 [42] in AV quality, semantic consistency, and AV synchrony. These gains largely stem from the explicit synchrony modeling through the spatio-temporal prior projector, coupled with a carefully designed training strategy. In addition, JavisGPT slightly outperforms the base generator of Javis- DiT [39], which we attribute to the stronger semantic understanding and encoding capacity of the LLM backbone [5]. leading more precise condition representations, ultimately improving the quality of generated sounding videos.
Interleaved comprehension and generation. As no existing benchmark directly measures joint under- standing and generation of sounding videos, we fill this gap by carefully curating a new evaluation set comprising 100 multi-turn QA dialogues covering anderstanding-generattt tasks. We employ five unbiased volunteers to con- duct human evaluation over several dimensions, cov- ering both textual and audio-video outputs. Full setup and detailed analysis are provided in Sec. E.1 The average results are reported in Fig. [6] where our method consistently outperforms the two compet- ing methods with joint comprehension-gent -eeeerrtnnnnneniinnnnnno capabilities by a large margin. Notably, we observe (1) NEXT- GPT [8T] frequently refuses to respond or generates noisy audio-vides outputs, severely impact- ing instruction-followin and generation quality; (2) Unified1O-2 42 is inferior to context reasoning and proactive thinking, failing to connect user history (e.g., preferences) with current instructions.
Pollowing
Unifi-to-4
NEXT.CPT
Javader
four types of interleaved
Quertion
Prooctive Thisking
Asswering
Geseration
Courext
Quility
Rezsoniig
Figure 6: Human evaluation on interleaved conversation. JavisGPT significantly outper- forms UnifiedIO-2 and NEXT-GPT.
6.2 Further Analysis
Effectiveness of SyncFusion on audio-video comprehension. To assess the impact of synchro- nization strategies, we train models solely on the AV understanding subset of JavisInst-Oma and evaluate their performance across three AV comprehension benchmarks. In Tab. [ we report the accuracy, the number of encoded AV tokens and per-sample inference latency. Results show that the interleaving strategy used Video-Salmonn2 [69] and Qwen2.5-Omm [86] fails to outperform simple concatenation [37 and introduces additional latency (over 2x slowdown) due to memory- discontinuous tensor operations. The Q-former
34 62 8T, despite offering faster inference, suffers
from optimization instability 22 5 and yields substantially lower performance. In contrast, our
8

---

Table 4: Comparing SyncFusion with other options for AV-synchronization. The SyncFu- sion module can effectively and efficiently cap- ture the audio-video synchrony.
Table 5: Ablation on training stages. We com- pare training without audio-video fine-tuning (w/o AV-FT), without multimodal pretraining (w/o MM-PT), and with all stages (w/ All).
AVSync
AVQA
MU-AVQA
AVSD
#Tokens]. Latency!
Quality
Consistency
Synchrony
Stage
Concal
93.3
80.7
61.3
3.5K
246ms
FVDJ
EAD
TV-IBt
TA-IR1 AV-IBt JavisScoref
Interleave
93.3
555ms
BiCrossAtn
93.2
80.6
61.1
3.5K
263ms
w/o AV-FT
473.2
9.6
0.252
0.089
0.152
0.069
Q-Fomee
71.4
54.7
56.3
0.2K
182ms
w/o MM-PT 306.7
8.4
0.267
0.132
0.178
0.135
SyncFusion
93.4
81.4
62.0
2,0K
224ms
w/ All
243.6
7.6
0.274
0.146
0.202
0.157
AV-Quility 1
Text- Consistemy
AV-Synchrony
Seaunk-Lial
Sepunto-0a:
Jvaa-L&G
Jain
673
72.5
Sipark
0.16
0.14
Telemass
2.5
0:37
0.34
0.10
23
2111
34277
0.06
1023
0.30
Suparate
Scouraic
2008 6000
5600
2000
$300
0000
3300
4000
6000
A-tad
AV-Ge--1
#lberition
#5teration
4treration
Figure 7: Comparison on joint and separate training of understanding and generation. Joint training generally leads to better perfor- mance, especially on the generation side.
Figure 8: Generation performance vs. training iterations. We observe that joint training con- sistently improves generation quality, instruction following (text consistency) and synchronization.
proposed SyncFusion achieves strong AV understanding performance while reducing both token count and inference latency, showing a more balanced trade-off between performance and efficiency.
Impact of the three training stages on high-quality AV generation. Tab, Janalyzs the contribution of each training stages to AV generation performance. We first evaluate the model after MM-Pretrain (Sec. [.1 without WV-FineTTnne and find that merely aligning with DiT's original text encoder [59] alone is insufficient to effectively bridge the LLM and DiT, resulting in poor generation quality (e.g., low JavisScore (39 of 0.069). Next, we skip pretraining and only apply diffusion loss for AV-Finetune (Sec. without MM-Pretrain). While this significantly improves generation quality, consistency, and synchrony, training exhibits instability in early stages. To ensure convergence, had to reduce the learning rate from le-4 to 1e-5, which ultimately compromises final performance. The best results are obtained by combining MM-Pretrain and AV-Finetune in our progressive training paradigm, confirming the necessity of staged adaptation for effective audio-video generation.
Joint training improves both understanding and generation. Fig. 7 compares the performance of comprchension and generation tasks under separate vs. joint training, with each score averaged over the corresponding test sets (e.g., 'V-Und' is the average accuracy over ActivityNet (6), Perception [53], and MVBench (35)). Results show that joint training consistently outperforms separate training in both understanding and generation tasks, indicating mutual enhancement effect. The improvement is more substantial on the generation side.
To better understand this effect, Fig. 8 presents the evolution of generation metrics across three dimensions. As the number generation training samples increases, the inclusion of comprehension data leads to steady gains in both AV-Quality and Text-Consistency. It suggests that semantic understanding of AV content enhances the quality of condition embeddings used for generation. This trend also aligns with findings in MetaQueries (51) for image tasks. However, AV synchrony shows only marginal improvement (+0.04), possibly indicating the need for a more unified mechanism bridging comprehension and generation. We further discuss this in Sec. A
instructioo-foolloeed and synchrony-aware comprehension and generation. Fig. presents a qualitative comparison with UnifiedlO-2 42) and NEXT-GPT [8I] in a two-round dialogue setting. In the first round, the models are asked to identify the source of gunfire sound. Unifiedl0-2 gives a general response ('a semi-automatic weapon'), while NEXT-GPT hallucinates
Case study:
explosion of a flower. In contrast, our model accurately grounds the sound video game scene, showing better spatio-temporal grounding and scene understanding, In the second round, the models asked

---

Discussion 2: ultimate framework for joint audio-video comprehension and generation. While LMMddrr-LLMdecoodl architectures with continuous inputs and diffusion decoders are well estab- lished 990] 81 42 unified end-to- and-to-end autoregressive models using both discrete and continuous Extending these from User: Watch the sounding video and answer the question:
What causes the gunfire sounds in the video?
UnifiedIO-2:
NEXT-GPT:
JavisGPT:
The gunfire sound in the video is probably caused automatic weapon.
The gunfire sounds video caused the explosive physical impact of flowers, which creates loud explosion.
gunfire sounds are produced by weapon, which part video game being played by the mun.
me video explosive physicul impact flowers loud intensive gunfire sounds.
Unified1O-2:
NEXT-GPT:
JavisGPT:
Ok, me suggest a vidco..
Certainly! have perfect vidco showcases this concept. It captures the flowers
No doubt, I will create vidco presence that reflects the flower gunfire.
Failed to generate
Figure 9: Case study on joint audio-video understanding and generation. (1) Round-I evaluates the instruction-baset AV understanding; (2) Round-II validates on context-based AV generation. JavisGPT can faithfully understand and generate high-quality synchronous audio-video content.
to generate gunfire-related video. UnifiedlO-2 produces irrelevant key frames, NExT-GPP fails to generate, while our JavisGPT successfully synthesizes coherent and synchronized audio-visual content. More examples are presented in Sec.
7 Conclusion and Discussion
propose JavisGPT, a unified MILM for synchronized audio-video comprehension and gen- eration. JavisGPT adopts an encoder-LLM-decodee architecture with SyncFusion module for spatiotemporal alignment and integrates a pretrained JavisDiT generator via hierarchical condition embedding. A three-stage training pipeline and the large-scale JavisInst- Ommi dataset enable com- plex audio-vidco- interactions. Extensive experiments demonstrate state-of-the-an performance on both unimodal and synchronized AV tasks, with interesting observations mutual enhancement on both understanding and generation capabilities. We believe JavisGPT offers a strong foundation for future research in synchronized media generation.
Discussion extended eapabilities for sounding video comprehension and generation. Our Javis- GPT can be further enhanced by: (1) Enabling speech input/output by replacing BEAT's with advanced models like Whisper or Wav Tokenizer [10] 57 86]; (2) Achieving fine-grained, controllable audio-video generation/editing via additional conditioning in JAV-DiT [29]; (3) Supporting complex instruction-basee generation through improved LLM reasoning and contextual conditioning [78).
tokens show strong synergy in understanding and generation [77 text-image to text toxt-audio-vie video modeling holds great promise.
Acknowledgement
This research/project is supported inpart by the National Research Foundation. Singapore, under its National Large Language Models Funding Initiative (AISG Award No: AISG-NMLP-2024-008 in part by Fundamental Research Funds for the Central Universities. and part by Zhejiang Provincial Natural Science Foundation of China under Grant No. LDT23FO101 and conclusions or recommendations expressed this material are those of the author(s) and do not reflect the views of the National Research Foundation, Singapore.
Any opinions, findings,
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint Xxix:23033.88744 2023.
[2] Huda AlAmri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K. Marks, Chiori Hori, Peter Anderson, Stefan Lee, and Devi Parikh, Audio visual scene-aware dialog. In Proceedings of the CVPR, pages 7558-7567, 2019.
[3] Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, et al. Audio visual scene-aware dialog. In Proceedings of the CVPR, pages 7558 7567, 2019.
[4] lean-Raptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch. Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Proceedings of the NeurIPS, 15523716--3775 2022.
[5] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge. Sibo Song, Kai Dang, Peng Wang. Shijic Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li. Jianqiang Wan, Pengfei Wang. Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
[6] Fabian Caba Heilbron. Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: A large- scale video benchmark for human activity understanding. In Proceedings of the CVPR. pages 961-970. 2015.
[7] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
[8] Honglic Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. Vggsound: A large-scale audio-vise visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). pages 721 725. IEEE, 2020.
[9] Jiuhai Chen, Zhiyang Xu, Xichen Pan, Yushi Hu, Can Qin, Tom Goldstein, Lifu Huang, Tianyi Zhou, Saining Xie. Silvio Savarese, al. Blip3-o: A family of fully open unified multimodal models-architeccure. training and dataset. arXiv preprint arXiv:250S.09566 2025.
[10] Sanyuan Chen, Yu Wu, Chengyi Wang. Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:22 2022.
[11] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pre pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501. 17811, 2025.
[12] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.
[13] Jiaxin Cheng, Tianjun Xiao, and Tong He. Consistent video-to-video transfer using synthetic dataset. arXiv preprint axXiv:2311.0022 2023.
[14] Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv preprint 2406v:2406.07777 2024.
[15] Jeongsoo Choi, Se Jin Park, Minsu Kim, and Yong Man Ro. Av2av: Direct audio-visual speech to audio-visual speech translation with unified audio-visume speech representation, In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 27325 22237, 2024.
[16] Sanjoy Chowdhury, Sayan Nag. Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, and Dinesh Manocha. Meerkat: Audio-visual large language model for grounding in space and time. In Proceedings of the ECCV, pages 52-70. Springer, 2024.
[17] Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang. Shiliang Zhang, Zhijie Yan. Chang Zhou, and Jingren Zhou. Qwen audio: Advancing universal audio understanding via unified scage audio languga models, arXiv preprint arXiv:2311.07919, 2023.

---

[18] Yunfei Chu. Jin Xu, Qian Yang. Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng. Yuanjun Lv, Jinzheng He. Junyang Lin, al. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759 2024.
[19] Konstantinos Drossos, Samuel Lipping. and Tuomas Virtanen. Clotha: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 736-740. IEEE, 2020.
[20] Hao Fei, Shengqiong Wu, Wei Ji. Hanwang Zhang, Meishan Zhang, Mong-Li Lee, and Wynne Hsu. Video-of-thought: Step-by-step video reasoning from perception to cognition. In Proceedings of the International Conference on Machine Learning, 2024.
[21] Hao Fei, Shengqiong Wu, Hanwang Zhang. Tat-Seng Chua, and Shuicheng Yan. Vitron: A unified pixel-level vision Ilm for understanding, generating, segmenting, editing. In Proceedings of the Advances in neural information processing systems, 2024.
[22] Hao Fei, Shengqiong Wu, Meishan Zhang, Min Zhang, Tat-Seng Chua, and Shuicheng Yan. Enhancing video-language representations with structural spatio-temporal alignment. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[23] Jort F Gemmeke, Daniel PW Ellis. Dylan Freedman, Aren Jansen, Wade Lawrence, Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human- dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal procesingg (ICASSP), pages 776-780. TEFE, 2017.
[24] Rohit Girdhar, Alaacldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all. In Proceedings of the CVPR, pages 15180-15190, 2023.
[25] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability lims via reinforcement learning, arXiv preprint ar5iv:2.01.1 /2948, 2025.
[26] Andrey Guzhov, Federico Rauc, Jörn Hees, and Andreas Dengel. Audioclip: Extending clip to image, text and audio. In /CASSP 2022-2022 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pages 976-980. IEEE, 2022.
[27] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. La Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. Proceedings of the ICLR, 1(2):3, 2022.
[28] Shengpeng Ji, Ziyue Jiang, Wen Wang, Yifu Chen, Minghui Fang. Jialong Zuo, Qian Yang. Xize Cheng, Zehan Wang, Ruiqi Li, et al. Wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling. In The Thirteenth International Conference on Learning Representations, 2024.
[29] Zeyinzi Jiang, Zhen Han, Chaojic Mao, Jingfeng Zhang, Yulin Pan, and Yu Liu. Vace: All-in-one video creation and editing. arXiv preprint arXiv:2503.0733 2025.
[30] Chris Dongjoo Kim, Bycongchang Kim, Hyunmin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 119-132, 2019.
[31] Dan Kondratyuk, Lijun Yu, Xiuye Gu, José Lezama, Jonathan Huang, Grant Schindler, Rachel Hornung, Vighnesh Birodkar, Jimmy Yan, Ming -Cang Chiu, al. Videopoet: a large language model for zero-shot video generation. In Proceedings of the 41st International Conference on Machine Learning, pages 25105-25124, 2024.
[32] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint 0rXiv.2408.03326 2024.
[33] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-inage pre-training unified vision-language understanding and generation. In Proceedings of the ICML, pages 12888-12900. 2022.
[34] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 22ip-2: Bootstrapping language- pre-train training with frozen image encoders and large language models. In Proceedings of the ICML, pages 19730-19999 19742. 2023.
12

---

[35] Kunchang Li. Yali Wang, Yinan He. Yizhuo Li. Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping. Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the CVPR, pages 22195-22206, 2024.
[36] Zhaowei Li, Wei Wang, YiQing Cai, Xu Qi, Pengyu Wang, Dong Zhang, Hang Song, Botian Jiang, Zhida Huang, and Tao Wang. Unifiedmllm: Enabling unified representation for multi-modal multi-tasks with large language model. arXiv preprint arXiv:240..02500 02503, 2024.
[37] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint
2023.
[38] Samuel Lipping, Parthasaarathy Sudarsanam, Konstantinos Drossos, and Tuomas Virtanen. Clotho- aqa: A crowdsourced dataset for audio question answering. In Proceedings of the EUSIPCO, pages 1140-1144.
2022.
[39] Kai Liu, Wei Li, Lai Chen, Shenggiong Wu, Yanhao Zheng, Jiayi Ji, Fan Zhou, Rongxin Jiang, Jiebo Luo, Hao Fei, et al. Javisdit: Joint audio-vides diffusion transformer with hierarchical spatio- prior synchronization. arXiv preprint wxXiv25003223777 2025.
[40] Xingchao Liu, Chengyue Gong, and Qiang Liu. Flow straight and fast: Learning to generate and transfer data with rectified flow. In Proceedings of the ICLR, 2023.
[41] Xiulong Liu. Zhikang Dong, and Peng Zhang. Tackling data bias music avqa: Crafting a balanced dataset for unbiased pnestion--nswweee
In Proceedings of the CVPR, pages
2024.
[42] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the CVPR, pages 26439 06439-26655, 2024.
[43] Simian Luo, Chuanhao Yan, Chenxu Hu, and Hang Zhao. Diff-foley: Synchronized vidco-to toudoo synthesis with latent diffusion models. Advances in Neural Information Processing Systems, 36:48855- 48876, 2023.
[44] Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang. Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv preprint 2:3..:2306.00000 09090, 2023.
[45] Yiyang Ma. Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie. Haowei Zhang, Liang Zhao, et al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation. arXiv preprint (idiv:2411..79777. 2024.
[46] Yuxin Mao, Xuyang Shen, Jing Zhang, Zhen Qin, Jinxing Zhou, Mochu Xiang, Yiran Zhong, and Yuchao Dai. Tavgbench: Benchmarking text to audible-video generation. Proceedings of the ACM MM, pages 6607-6616, 2024.
[47] Irene Martin-Morate and Annamaria Mesaros. What is the ground truth? reliability of multi-annotatoo data for audio tagging. In 2021 29th European Signal Processing Conference (EUSIPCO), pages 76-80. IEEE, 2021.
[48] Xinhao Mci, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. Wavcaps: A chatgpt- assisted weakly- audio captioning dataset for langoela multimodal rescarch. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024.
[49] Annamaria Mesaros, Toni Heittola. Aleksandr Diment, Benjamin Elizalde, Ankit Shah, Emmanuel Vincent, Bhiksha Raj, and Tuomas Virtanen. Dease 2017 challenge setup: Tasks, datasets and baseline system. In DCASE 201 wookkshho on detection and classification of acoustic scenes and events, 2017.
[50] Microsoft, :, Abdelrahman Abouelenin. Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishray Chaudhary, Congcong Chen, Dong Chen, Dongdong Chen, Junkun Chen, Weizhu Chen, Yen-Chun Chen, Yi ling Chen, Qi Dai, Xiyang Dai, Ruchao Fan, Mei Gao, Min Gao, Amit Garg, Abhishek Goswami, Junheng Hao, Amr Hendy, Yuxuan Hu, Xin Jin, Mahmoud Khademi, Dongwoo Kim, Young Jin Kim, Gina Lee, Jinyu Li, Yunsheng Li, Chen Liang, Xihui Lin, Zeqi Lin, Mengchen Liu, Yang Liu, Gilsinia Lopez, Chong Luo, Piyush Madan, Vadim Mazalov, Arindam Mitra, Ali Mousavi, Anh Nguyen, Jing Pan, Daniel Perez-Becker, Jacob Platin, Thomas Portet, Kai Qiu, Bo Ren, Liliang Ren, Sambuddha Roy, Ning Shang, Yelong Shen, Saksham Singhal, Subhojit Som, Xia Song, Tetyana Sych. Praneetha Vaddamanu, Shuohang Wang. Yiming Wang. Zhenghao Wang, Haibin Wu, Haoran Xu, Weijian Xu, Yifan Yang. Ziyi Yang. Donghan Yu, Ishmam Zabir, Jianwen Zhang, Li Lyna Zhang, Yunan Zhang, and Xiren Zhou. Phi-4-min technical report: Compact yet powerful multimodal language models via mixture-of loras, 2025.
13

---

[51] Xichen Pan. Satya Narayan Shukla, Aashu Singh. Zhuokai Zhao, Shiok Kumar Mishra, Jialiang Wang- Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, et al. Transfer between modalities with metaqueries. arXiv preprint arXiv:2504.06235 06256, 2025.
[52] Se Jin Park, Chae Won Kim, Hyeongseop Rha, Minsu Kim, Joanna Hong. Jeong Hun Yeo, and Yong Man Ro. Let's go real talk: Spoken dialogue model for face-to-face conversation. arXiv preprint arXiv:2406.07865 2024.
[53] Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang. Carl Doersch, et al. Perception test: A diagnostic benchmark for multimodal video models. Proceedings of the NeurlPS, 36:42748 2023.
[54] Karol J Piczak. Esc: Dataset for environmental sound classification. In Proceedings of the 23rd ACM international conference on Multimedia, pages 1015-1018, 2015.
[55] Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lec, Apoorv Vyas, Bowen Shi, Chih-Yar Ma, Ching-Yar Chuang, et al. Movie gen: A cast of media foundation models. arXiv preprint 24X::22410.1370000 2024.
[56] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from PmLR, 2021.
natural language supervision. In Proceedings of the ICML, pages
[57] Alcc Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on machine learning. pages 28492-28518. PMLR, 2023.
[58] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 3e0.:728:33778318303 2023.
[59] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 21((40))1-67, 2020.
[60] Ludan Ruan, Yiyang Ma, Huan Yang. Huiguo He, Bei Liu, Jianlong Fu, Nicholas Jing Yuan, Qin Jin, and Baining Guo. Mm-diffusion: Learning multi-modal diffusion models for joint audio and video generation. In Proceedings of the CVPR, pages 09219110228 2023.
[61] Justin Salamon, Christopher Jacoby, and Juan Pablo Bello. A dataset and taxonomy for urban sound research. In Proceedings of the 22nd ACM international conference on Multimedia, pages 1041-1044, 2014.
[62] Paul Hongsuck Seo, Arsha Nagrani, and Cordelia Schmid. Avformer: Injecting vision into frozen speech models for zero-shot av-asr. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 22922-2293 2023.
[63] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 2xXxx:2402.03300 2024.
[64] Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face. Proceedings of the NeurlPS, 2023.
[65] Fangxun Shu, Lei Zhang, Hao Jiang, and Cihang Xie. Audio-visual 1l for video understanding. arXiv preprint 2tl:::2312.062222 2023.
[66] Bob I. Sturm. The gtzan dataset: Its contents, its faults, their effects on evaluation, and its future use. arXiv preprint / 1461, 2013.
[67] Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction- follow them all. arXiv preprint arXiv:2305. 16355, 2023.
[68] Guangzhi Sun, Wenyi Yu, Changli Tang, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma. Yuxuan Wang, and Chao Zhang, iideo-salmonn Speech-enhaneee audio-visual large language models, arXiv preprint arXiv:2466.11 15704, 2024.
14

---

[69] Changli Tang, Wenyi Yu, Guangzhi Sun, Xianzhao Chen, Tian Tan, Wei Li, Lu Lu, Zejun Ma. and Chao Zhang. Salmonn: Towards generic hearing abilities for large language models. arXiv preprint arXiv:330.13288, 2023.
[70] Changli Tang, Yixuan Li, Yudong Yang. Jimin Zhuang, Guangzhi Sun, Wei Li, Zujun Ma, and Chao Zhang. Enhancing multimodal I1m for detailed and accurate video captioning using multi-round preference optimization. arXiv preprint arXiv:2410.06622 2024.
[71] Zineng Tang, Ziyi Yang. Chenguang Zhu, Michael Zeng. and Mohit Bansal. Any-to-any generation via composable diffusion. Proceedings of the NeurlPS, 36:16083- 16:16083-16099 2023.
[72] Zineng Tang, Ziyi Yang, Mahmoud Khademi, Yang Liu, Chenguang Zhu, and Mohit Bansal. Codi-2: In- context interleaved and interactive any-to-any generation. In Proceedings of the CVPR, pages 27425-27434, 2024.
[73] Chamelcon Team. Chamelcon: Mixed-modal carly-fusion foundation models.
arXiv preprint
S.Kv:2200500888.. 2024.
[74] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du Chonghua Liao, et al. Kimi kl. S: Scaling reinforcement learning with Ilms. arXiv preprint wxxv:2501.12599 2025.
[75] Shengbang Tong. David Fan, Jiachen Zhu, Yunyang Xiong, Xinlei Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint arXiv:2412. 14164, 2024.
[76] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution, arxiv preprint arXiv:2409. 2024.
[77] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang. Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction all you need, arXiv preprint wwXiv:2409.18866 2024.
[78] Cong Wei, Quande Liu, Zixuan Yc, Qiulin Wang, Xintao Wang. Pengfei Wan, Kun Gai, and Wenhu Chen. Univideo: Unified understanding, generation, and editing for videos. arXiv preprint arXiv:2510.00 083777. 2025.
[79] Shengqiong Wa, Hao Fei, Hanwang Zhang, and Tat-Seng Chua. Imagine that! abstract-to-intricat text-to-image synthesis with scene graph hallucination diffuusion. In Proceedings the 37th International Conference on Nearal Information Processing Systems, pages 79240-79259, 2023.
[80] Shengqiong Wu, Hao Fei, Xiangtai Li, Jiayi Ji, Hanwang Zhang, Tat-Seng Chua, and Shuicheng Yan. Towards semantic equivalence of tokenization multimodal 1im. arXiv preprint arXiv:2406.05122 2024.
[81] Shenggiong Wu, Hao Fei, Leigang Qu. Wei Ji, and Tat-Seng Chua. NEXT-GPT: Any-to-any multimodal LLM. In Proceedings of the ICML, pages 53366-53397, 2024.
[82] Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint ar Xi:: 240,,0499 2024.
[83] Yusong Wu Ke Chen Tianyu Zhang*. Yuchen Hui*, Taylor Large-scall contrastive language- audio pretraining with feature fusion and keyword-to- captiio augmenta- tion. IEEE International Conference on Acoustics, Speech and Signal Processing, ICASSP, 2023.
and Shlomo Dubnov.
[84] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu. Zhijie Chen, Zhenheng Yang. and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
[85] Zhifei Xie and Changqiao Wu. arXiv preprint arXiv:2408. 16725, 2024.
Language models can hear, talk while thinking in streaming.
[86] Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang. Yang Fan, Kai Dang, et al. Qwen2. 5-omni technical report. arXiv preprint 2503.222002222555 2025.
[87] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint 251 2024.
15

---

[88] An Yang, Anfeng Li, Baosong Yang. Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et. Qwen3 technical report. arXiv preprint urXiv:2505.09388 2025.
[89] Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. Avqa: A dataset for audio-visual question answering videos. In Proceedings of the ACM MM, pages 3480-3491, 2022.
[90] Jun Zhan, Junqi Dai, Jiasheng Ye, Yunhua Zhou, Dong Zhang, Zhigeng Liu, Xin Zhang, Ruibin Yuan, Zhang, Linyang et al. Anygpt: Unified multimodal Ilm with discrete sequence modeling. arXiv preprint XxXiv:2402.12220 2024.
[91] Hang Zhang, Xin and Lidong Bing. Video-Ilama: instruction tuned audio-visual language model for video understanding. arXiv preprint vrXiv:2306.028555 2023.
[92] Tianyi Zhang, Varsha Kishore, Felix Kilian Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv1994.006755 2019.
[93] Y Zhang, Li, H Liu, Lee, Gui, D Fu, Feng, Z Liu, and C Li. Llava-next: A strong zero-shot video understanding model. arXiv preprint arXiv:2311. 2024.
[94] Yiming Zhang, Yicheng Yanhong Zeng. Zhening Xing. Yuancheng Wang, Zhizheng Wu, and Kai Chen. Foleycrafter: Bring silent videos to life with lifelike and synchronized sounds. arXiv preprint urXiv:2407.01494. 2024.
[95] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma. Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713, 2024.
[96] Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-mode modal model. arXiv preprint arXiv:2408. 11039, 2024.
16

---

In the appendix, we present discussion in limitation and societal impact (Sec. A details in model implementation and evaluation (Sec. B. construction details of JavisInst- Omii (Sec. C.. discussion on architecture designs (Sec. [ supplementary investigations on JavisGPT (Sec. and more quantitative case studies on JavisGPT (Sec. FD.
A Potential Limitation and Societal Impact
A.1 Limitation and Future Work
While our JavisGPT represents a pioneering effort in synchronized audio-video understanding and generation, the field itself is still in its early stages. In this section, we discuss the current limitations of our method within the context of its time and outline several directions for future work.
Architecture. JavisGPT adopts a straightforward encoder-LLM-decodee architecture. While effective, as briefty mentioned in Sec. this design presents core inconsistencies: (1) Misaligned training objectives. The comprehension tasks rely on next-token prediction (NTP) loss [5] for textual outputs, whereas the generation tasks use diffusion loss [39] (with alignment loss) for audio- visual outputs. These distinct objectives lead to inconsistent optimization signals for the shared LLM backbone. (2) Asymmetric iuput-output modeling. For multimodal content (in this case, audio and video), the input relies on VAE-based continuous embeddings [S), while the output is conditioned through query-based interface [51] The connection between the two indirect: the generation queries can "see" the embeddings from the comprehension stage, allowing understanding to enhance generation; however, the reverse is not true, as the generation queries are not part of the comprehension input. To achieve a truly unified MLLM for audio-video understanding and generation, it may be beneficial to draw inspiration from recent advances in unified modeling for image-based multimodal tasks 777 82 75 and adapt them to the audio-video domain. Sec. Dec.Dfurttin further elaborates on possible architectural directions for unifying comprehension and generation, as well as improving spatiotemporal synchronization, which we hope can provide valuable insights for future research in the community.
Scalability. For the sake of rapid development and verification, our current implementation is built upon 7B-scale backbone LLM [5] and trained on a limited collection of publicly available datasets. As such, the scalability of our approach has not been fully explored. With access to larger computational resources, scaling the backbone LLM to 70B-size or beyond, along with training on trillions of multimodal tokens, may lead to significant performance improvements and unlock further potential in joint audio-video modeling.
Aligument. In the current post-training stage, JavisGPT is only intrucction-tunee to acquire basic cnatuuction-oollwinn and simple reasoning capabilities. To further enhance the model's generalization ability, reinforcement learning (RL) presents promising direction for post-training. Specifically: (1) On the comprehension side, RL can transform the base JavisGPT into thinking model 25
74 88
capable of handling complex reasoning tasks in real-world scenarios. (2) On the generation side, RL can substantially improve the quality, textual consistency, and audio-video synchrony of the generated We leave this direction as an important avenue for future exploration.
content
58 63
A.2 Societal Impact
JavisGPT enables synchronized audio-vides video comprehension and generation with language, supporting applications in education, accessibility, and interactive media. However, the same capabilities also raise important societal concerns. The model could be misused synthesize misleading or harmful audio-visual content, contributing to the spread of misinformation or impersonation (c.g., deepfakes). If trained on sensitive unfiltered data, it may also reproduce biased private content, leading to fairness and privacy risks. Moreover, as generation models become more accessible, the barrier to misuse may decrease.
To mitigate these risks, responsible deployment practices are essential, including controlled access (e.g., through APIs), content watermarking, dataset auditing, and development of detection tools. While JavisGPT is released as research prototype, we highlight the need for continuous evaluation of its broader impact the field progresses.
17

---

B JavisGPT Model Details
B.1 Model Architecture
We propose JavisGPT, a general framework that transforms a vision-language MILM backbone into a unified model capable of joint comprehension and generation of sounding videos. The backbone LLM is flexible and model-agnostic, allowing for various architectures and scales. In the manuscript, we adopt Owen2.5-VL-7BB-nnsti Instruct [5] as our primary backbone, and this section details the architecture configuration. Investigation on alternative base LLMs is also provided in Sec.
Backbone MLLM. Qwen2.5- Sive2..5VV-77B follows decoder-only architecture, consisting of 28 Trans- former blocks with a hidden size of 3584. During training, the original model parameters are kept frozen. We inject trainable LoRA 27 modules into each linear layer, with hyperparameters set to r 128 and a 256. After training, the LoRA weights can he merged back into the LLM via reparameterrization, incurring additional overhead during inference.
Visual Encoder and Projector. We adopt the ViT and Merger modules from Qwen2.5-VL [ as the visual encoder and projector, respectively, and keep their parameters frozen during training. The ViT has a hidden size of 1280 and a patch size of 14, After feature extraction, the Merger applies a 2x2 patch grouping and projects it into the LLM token embedding space with a dimension of 3584.
Audio Encoder and Projector. We adopt BEATs [10 as the audio encoder and keep it frozen during training. A learnable 2-layer MLP is used as the projector to map the 768- 388-dimensional features produced by BEAT's to the 3584-dimensional token embedding space of the L.I.M. For all newly introduced MIPs in this work, the intermediate size is set as 4x of the output embedding size, and we omit this detail in the subsequent descriptions.
Audio-Vides Video Synchronizer (SyncFusion) It consists of an MHSA layer with a hidden size of 3584 and 8 attention heads, with the FFN also implemented as a simple 2-layer MLP.
Audio-Vidie Generator and Projector. We use JavisDiT-vO.1 39 as the DiT backbone, which consists of 28 layers of dual-stream DiT blocks with a hidden size of 1152. The text encoder is T5- XXL [59], with an embedding size of 4096 and a context length (e.g.. number of learnable semantic queries) of 300. The spatiotemporal prior encoder is ImageBind (24). with an embedding size of 1024 and a context length (e.g., number of learnable spatiotemporal prior queries) of 77. Both the semantic projector and the spatiotemporal prior projector are 2-layer MIPs, mapping the 3584-dime LLM hidden states to 4096- and 1024-dimensional condition embeddings. respectively.
B.2 Training Configuration
Building upon the strong visual understanding and basic instruction-followe capabilities of the Qwen2.5-VI. VL. [5] backbone, we adopt the three-stage training strategy described Sec. S develop JavisGPT for joint understanding and generation of synchronized sounding videos. Table Al sum- marizes the key configurations, and the training and data details are provided below. All models are trained on 8 NVIDIA A100-80GB GPUs.
This stage aims to equip the model with basic audio-video understanding and generation capabilities. (1) Understanding: We use 600K audio-text pairs to enhance the audio understanding ability. Tasks include audio captioning and question answering, with data sourced from AudioSet (23) AudioCaps (30], VGGSound [8). WavCaps [48), Clotho [19] ESC50 (44 GTZAN (66), MACS (47, and UrbanSound8K (61). Training conducted using next-token prediction loss. (2) Generation: We use I 1.5M vidfiovvidoocatttinnn triples from TAVG- Bench [46] to pretrain the generation component, using caption alignment loss Latign only. During this stage, the trainable components include the audio projector, query embeddings, and condition projector. We set the learning rate to 1c-3 and train for one epoch.
Stage Multimodal Pretraining.
Stage H: Audio Video Fine Tuning. This stage aims to enhance the model's ability to capture and maintain synchronization between audio and video in both understanding and generation tasks. (1) Understanding: We use 450K audio-video-ter text triplets from TAVGBench [46] and train on the sounding video captioning task using nexi-token prediction loss. (2) Generation: The same 450K triplets are reused for ext-to-sounding-vv diffusion loss. this stage, we additionally fine-tune the LoRA modules of the L.M,, along with previously trainable components. The learning rate is set to Ic-4, and training runs for one epoch.
generation, with a combination of caption loss and
18

---

Table A1: Detailed settings for progressive audio--idee-synchrooizee training.
Setting
Stage-I MM-PreTrain A-Und AV-Gen-Pre
Stage-II
Stage-III
Purpose
AV-FineTune
MM-Ins(Tune
Tasks
AV-Und AV-Gen
A/V/AVUUnd AV-Gen
Trainable Module
8" + +Q , .00
Ba + Q " 6 + Viora
+ ,00 + Voora
Trainable Params
239.95M
654.68M
717.09M
Training Objective
Cutp + Latign 600K + 1.5M
Lutp + Latign + Lass
Cutp + Latign + +aaf 600K
Training Samples
450K 450K
Training Epochs
1
Warm-up Epochs
0.03
0.03
0.03
Batch Size
256
64
64
Learning Rate
lc-3
lc-4
lc-4
Weight Decay
0.0
0.0
0.0
GPU Days (A100)
6.5
14.2
9.3
Stage III: Multimodal Instruction Tuning, This stage is designed to equip the model with instrutiion-ollooiing and basic reasoning capabilities. (I) Understanding: We first construct the JavisInst-Uni Und subset, a synchrony-aware audio-video QA dataset spanning three levels: entity, relation, and global comprehension. consists of 110K samples derived and processed from TAVG- Bench [46]. Details can be found in Sec. To further enhance performance, we also incorporate 95K audio-video understanding samples from VideoLLaMA2 [14], including training splits from AVQA 89] MusicAVQA [1], and AVSD 3
To prevent catastrophic forgetting, we additionally include 20K image understanding samples from VLaVA--ne Vision 32), 60K video understanding samples from LLa Video- 178K [95], 550K audio comprehension samples from Stage I's dataset, and 20K audio-video caption samples from TAVGBench [46). (2) Generation: Since no existing AV-instruction dataset is available for generation tasks, construct the JavisInst-Gen subset based on repurposed data from TAVGBench. It covers diverse set tasks, including text-to-audio- video generation, in-context generation, and interleaved multi-tun conversation. More details provided in Sec. [ Besides, 1 150K audio-video generation instances are also included to maintain JAVG capability learned from the previous stage.
B.3 Evaluation Setup
In Sec. 6 we comprehensively evaluate the multimodal understanding and generation performance of JavisGPT, and here we introduce the detailed evaluation setups.
Multimodal comprehension. We follow the evaluation protocol of VideoLLaMA2 [4], and conduct unified benchmarking across visual understanding, audio understanding, and synchronized audio- video understanding for various multimodal large language models. The evaluation covers open-ended and multiple-chi choice formats, with single- multi-turn dialogues. (1) vi deo understanding tasks are evaluated on three subsets: AetivityNett-QA (6), Perception Test [53], and MVBench [35]; (2) audio understanding is evaluated understanding includes AVQA 189) MusicAVQA [11. and AVSD 3. All models are evaluated in the zero-shot setting using greedy decoding ensure reproducibilityy Following Lide (14),
Clotho-AQA [8] and TUT2017 [9); and (3) audio-video
we report the average accuracy as the primary evaluation metric. For multiple-choice datasets, accuracy calculated via lexical match, while open-ended responses are evaluated with the assistance of GPT-3.5 []] To ensure fair comparison, input videos are uniformly sampled to 16 frames, and audio preserved at 16 kHz across all models.
Multimodal generation. For audio-vider generation evaluation, we follow the protocol of Javis- DiT (39), using 1,000 text-to-auee audio-vided samples from JavisBench-min. [39 for comprehensive assessment. All models are evaluated with 4-second videos at 240P resolution and 24fps, and audio sampled at 16kHz.
comprehensive evaluation of quality, consistency, and synchrony for audio-video generation results is provided. Here, we briefly introduce the mechanisms of each evaluation dimension:
19

---

Table A2: Comparison on sounding video datasets for instruction tuning.
Dataset
Quantity
Diversity
AV-Synchrony
Multi-Turn
Reasoning
Comprehension
MusicAVQA 41
32K
X
X
AVQA 89
40K
V
X
X
AVSD 3
8K
V
X
X
JavisInst-Und (Ours)
110K
V
Generation
MosIT R1
5K
x
X
JavisInst-Gen (Ours)
90K
Audio / Video Quality: measuring the perceptual quality of the generated audio and video, including (1) Fréchet Video Distance (FVD): FVD = ||,,- Ma 1 + Tr(2, -2, 2 where (Kr, ) and (4gg2ig) are the mean and covariance of ground-truth and generated video features extracted by a pretrained I3D encoder (7]. Lower is better, indicating the generated video distribution is closer to the real one; (2) Kernel Video Distance (KVD): similar to FVD, but estimates distribution differences via a kernel-based method (Kernel Inception Distance style), which is more stable smaller datasets; lower is better; and (3) Fréchel Audio Distance (FAD): same concept as FVD, but computed on audio features extracted by a pretrained AudioClip model 266 measuring distribution distance between generated and real audio; lower is better.
Text Consistency: evaluating how well the generated audio and video semantically match the input text description, including (1) ImageBind (24) text-video cosine similarity: sim(t, v) (t)-S(t)fviiee
[fle [v]T' (2) ImageBind text-audio cosine similarity: same process but with the audio encoder faudio: (3) CLIP-Score: using CLIP [56] compute semantic similarity between text and video (video frames are sampled, encoded, and averaged); and (4) CLAP-Score: using CLAP 88
compute semantic similarity between text and audio.
Audio-Vides Semantic Consistency: measuring the semantic alignment between generated audio and generated video, including (1) ImageBind audio-video cosine similarity, encoding both modalities into the same space and computing cosine similarity between video and audio
features; and (2) Audio- Visual Harmony Score (AVHScore): introduced in TAVGBench [46] as a way to quantify how well the generated audio and video align semantically in a shared embedding space. It is defined by computing the cosine similarity between each video frame and the entire audio, then averaging across all frames: AVHScore
Σ cos frrame (e:), higher AVHScore indicates stronger audio-vided semantic consistency. Note that we remove the CAVP-
(a)). A
4 used in JavisDiT (39 because this metrie keeps range from 0.798 to 0.801 and cannot capture the difference when evaluating semantic consistency.
Audio-Video
Synchrony: evaluating spatiotemporal alignment in generated audio-video pairs, focusing on JavisScore: new metric proposed in JavisDiT (39 The
idea is to use a sliding window along the temporal axis to split the audio-video pair into short segments. For each segment, compute cross-modal similarity with ImageBind and take the mean
score: JavisScore E (a, v;), (l,,, = ビ top-k f coos (vi,)) (a:)))
min
C JavisInst-Omni Dataset Details
C.1
Motivation
To advance instruction tuning for multimodal large language models in sounding video scenarios, we introduce two new datasets: JavisInst-Und and JavisInst-Gen, which support open-ended QA for wide range of understanding and generation tasks, respectively. This motivated by our observation that existing datasets suffer from significant limitations in terms of scale, capability coverage, and task diversity. Tab. A2 summarizes the comparison with existing mainstream datasets. Below, we highlight the key differences and contributions of our proposed datasets.
20

---

Audio-video comprehension. While AVQA 89 MusicAVQA 4T, and AVSD B provide some diversity or dialogue support, they mainly fall short in modeling and evaluating audio-video synchrony (AV-Synchrony). Specifically, MusicAVQA Al is limited to musical scenarios and lacks content diversity. AVQA [89] covers a wide range of everyday scenes, but the majority of its questions are simple existential queries such as "what" or ""wher",, with little emphasis on event-level synchrony between audio and video, and it does not support multi-turn interactions. AVSD addresses the multi-turn dialogue limitation, but still does not emphasize AV-synchrony, and its relatively small scale (8K samples) makes it unsuitable for large-scale training. To address these limitations, introduce JavisInst- Und, which not only inherits the strengths of prior datasets but also emphasizes on synchrony- aware comprehension and expands the scale to 1 10K samples, providing a solid foundation for large-scale synchronized audio-vided understanding.
[89] partially
Audio-video generation. In fact, there currently exists no instruction dataset specifically designed for synchronized audio-video generation. Although the MosIT dataset (see Tab. (22 includes multi- turn dialogues and reasoning capabilities, it does not contain direct training samples for generating audio-video content. To fill this gap, we introduce JavisInst-Gen a wide range of everyday conversational scenarios. It supports synchronized AV generation with multi-turn interaction and reasoning capabilities, addressing the current limitations in the field.
dataset of 90K samples covering
JavisInst- Und and JavisInst-Gees are primarily constructed from TAVGBench [46] and InstV2 We leverage GPT-4o [ and video-to rideo-to-audio tools [94] for QA generation and data processing, and Fig. A3 showcases some representative examples for multi-tunn comprehension and generation.
C.2 JavisInst-Und
Taxonomy. As detailed in Tab. A3 we construct a hierarchical taxonomy that captures information from local to global levels, covering 3 major categories and 10 subcategories: (1) Entity-level, including existence, alignment, grounding, and counting tasks, aiming to capture fine-grained auditory and visual characteristics of each individual sounding event. (2) Relation-level, includes spatial, temporal, and causal relationships in modeling the spatiotemporal and causal interactions between different sounding events. and (3) Global-level, including theme, emotion, and atmosphere analysis, aiming to capture the overall emotion and thematic expression of audio-video content.
Construction. We propose an efficient method for constructing JavisInst-Und by reusing audio- video samples from AudioSet [3] and leveraging the corresponding text captions provided in we generate diverse QA TAVGBench [46). With the powerful language capabilities of GPT-4o I pairs for audio-video understanding.
Single-turn audio-video understanding: For each of the 10 predefined dimensions, we design prompt templates (as exemplified in Fig. Al] and adopt few-shot prompting strategy guide GPT-4o in generating eategory-specifie QA pairs based on the given audio- video caption. We randomly generate QA exercises from 3-4 categories for each sounding video caption.
Multi-turn audio-video understanding: For videos from the same source in TAVGBench, we construct multi-turn dialogues in two ways: (1) randomly compositing multiple single-turn QA samples into a multi-turn session ("Composite")): and (2) first curating a single-turn instruction- based audio-vide video generation dialogue (see Sec. C.3), and then augmenting it with the corresponding single-turn QA samples above to create 2-3 turn "Gen2Und" conversations, further improving scene diversity.
In total, we construct a dataset of 110K QA samples in JavisInst- Und (see Fig. 5 for distribution), supporting a broad range of audio-video understanding tasks and applications.
C.3 JavisInst-Gen
Taxonomy. Table provides a detailed overview of the three types of audio-video generation tasks included our dataset, along with representative examples: (1) Instruction based Generation. This is the core text-to-audio-i -ideo generation task. We distinguish between formal-style (more written and detailed) and colloquial-style (more casual and concise) instructions to better reflect real-world dialogue scenarios. (2) Conditional Generation. This task aims to generate new audio-video content based on modalities beyond text (e.g., images, videos, audio, or any combination thereof) as a context condition. It broadens application potential and includes tasks such as audio-video extension for long-
21

---

Table A3:
Clarification of the category faxonomy of JavisInst-Und.
Aspect
Category
Description
Example (explanation omitted)
Existence
Whether an object with specific audio characteristic appears in the vidco.
Que: Is the cat meowing? Ans: No.
Alignment
Whether the object making a certain
Que: Does lion video make the
sound and the object the vidco are the same one.
roaring sound the audio? Ans; Yes.
Entity
Grounding
The position of the object with certain
Que: Where is the dog that barking?
audio characteristic the video.
Ans; Near a red
Counting
The number of some objects in the audio
Que: How many people can be identi-
and video.
fied in the video and audio? Ans: Two.
Spatial
The relative position of the different ob-
Que: Where is the barking dog in rela-
jects that make the sounds.
tion the cat that meowing? Ans:
Over the
Temporal
The temporal relationship hetween the audio and video.
Que: Dñes the honking sound appear
Relation
before the pedestrian crosses the street?
Ans: Yes.
Causal
The activity relationships hetween dif-
Que: What causes the lond crash sound?
ferent objects.
Ans: The child dropping glass.
Emotion
The emotions expressed by the charac-
Que: What emotion is the woman
uppearing in the video.
pressing? Ans: Happy.
Atmosphere
The common atmosphere in the video
Que: How about the atmosphere shown in the video and audio? Ans: Chaotic.
Global
and audio.
Theme
Whether an object with a specific audio characteristic appears in the video.
Que: What is overall theme the video? Ans: Family picnic.
Composite
2-3 rounds of conversations about the
Que: Where is the source of the ma- chine gun firing sound? Ans: From the helicopier. Que: When does the sound of the machine gun firing occur? Ans: While the helicopter is flying.
same sounding video.
MultiTurn
Gen2Und
Model first generates a sounding video, then answers a corresponding question.
Que: Generate video that follows these guidelines: as the boat tow truck pulls Ans: Sure! Que: Does the boat heing lifted out of the water produce any sound in the audio? Ans: No.
form video generation and moars--grained AV editing. (3) Multi-turn Conversation. This includes more complex task setups involving proactive reasoning and beteesstnd-then-geeeee (Und2Gen) scenarios, where the model must first comprehend prior context before generating. These tasks further enrich the diversity and complexity of the datasct.
Construction. JavisInst-Gen is also efficiently constructed by organizing data around the three defined types of audio-video generation tasks:
Instruction based Generation: This is the core task, where the model is required to generate synchronized audio-video content conditioned
given text caption. To support this, we leverage GPT-4o [ to construct 3,000 diversified prompt templates, and randomly sample 45K audio-video captions from TAVGBench [46] combine with these templates, forming a wide range of instruction- based generation instances. To enhance linguistic diversity, approximately 20% of the instructions are further paraphrased by GPT-44-mini (for cost efficiency), making them more aligned with natural, conversational instructions typically found in real-world applications.
This task follows similar construction strategy to instruction- gen- eration, where we synthesize a large number of conditional generation cases by randomly combining captions, followed by modality-specinf input processing.
Conditional Generation:
prompt templates with existing audio- For example, in A2V tasks, the audio is extracted as input; and in 12AV, the first frame of the video is used as input. For AV-Extension, sample two overlapping clips (e.g., 1-2s overlap) from ios vidco in TAVGBench, using the first input and the second the generation target. As for
22

---

Prompt for Spatial Relationship Question-Answering Data Synthesis
# Instruction
Given a detailed auido-video joint description that summarizes the content of a audio and video, generate multi-choice question-answer exercises based the description help humans better understand the audio and video.
Category: Spatial Relationship
In this dimension, you need to focus on the relative position of the different objects that make the sounds. The question can be formatted as: (1) "Where the <object that produced <soundI> in related to the <object2> that made <sound2>?", (2) "Is the <objectl>
behind <object2> that makes <soundl>?", (3) "Which <object> the nearest to <object2> that makes <sound2>", etc. If there is NO exact description of the relative position of the sounding objects in the caption, output NO.
# Examples
## Caption: A lion roars while standing rock in the middle of savanna. i"Question": "Does the lion in the video make the roaring sound in the audio?". "Options": ("A": "Yes", "B": "No"), "Answer": "A", "Explanation": "The audio aligns with the visual of the lion roaring in the savanna.") ## Caption: The mountains
covered with lush forests and variety of flowers, with the faint sound of flowing water in the distance.
""Question": "Does the object producing the water flowing sound appear in the video?", "Options": ("A": "Yes", "B": "No"). "Answer": "B", "Explanation": "The video only shows mountain with forests and flowers, not the source of the water flowing sound.") ## Caption: The man is shown fishing in river, holding fishing rod in his hand while fish can be seen swimming in the water. I"Question": "NO", "Answer": "NO")
# Target
Now generate a question-answer sample according to the given caption:
## Caption: (caption)
Figure A1: Prompt for spatial relationship question-answering data synthesis. Here (caption) is the placeholder for given audio-video caption.
Prompt for Proactive Audio- Video Generation Data Synthesis
Please extract an unimportant entity or concept (such as an object, place, or activity) from the given text and perform the following tasks:
Remove the extracted concept from the text while keeping the original meaning intact.
2. Generate sentence expressing human preference
3. Generate a natural response to the preference statement.
Output Format: Please output the result in JSON format with the following keys:
'modified_teww text': The text after removing the extracted concept.
"generated preference': The sentence expressing human preference.
rresponse: The response to that sentence.
The given text is: [caption]. Directly output the rewritten content without any additional text.
Figure A2: Prompt for proactive audio-video generation data synthesis.
AV-Edit, since TAVGBench (and other related datasets) lack suitable editing instructions, we adopt an alternative approach: randomly sample 5K video editing pairs from InstV2v and apply
23

---

Table A4: Clarification of the category taxonomy of JavisInst-Gen.
Task
Category
Description
Example (response omitted)
Formal
Sounding video generation with formal instruction.
Que: Compose thorough video piece featuring narration and imagery. The sound of motorcycle engine
Instruct
Colloquial
Sounding video generation with collo- quial instruction.
Que: Make this into a video right away.
A female vocalist is performing
V2A
Generate an audio for input video.
Que: Mount audio track the visu-
als. The intense and fast fast-paced
A2V
Generate video for input audio.
Que: Produce video narrativ::
robotic vacuum cleaner parked
A12V
Animate an image with input audio.
Que: Make a video with the picture and
soundtrack. A motorcycle revving.
Condition
12AV
Animate an image and make an audio.
Que: Compose a video from this image,
with sound. As the toy police car sits
AV-Ext
Temporally extend the sounding video.
Que: Continue from the stopping point.
A man wearing a pink t-shirt
AV-Edit
Edit the sounding video supposed.
Que: Make the concept art look like
cient Egyptian hieroglyphs.
Proactive
Model first extracts user preference, then generates a corresponding sound- ing video.
Que: What do you think about oranges as fruit? Aus: They're re really invigo- rating and full of flavor! Que: Create a video where car driving... with fruit balanced on top of it.
Rethink
Model first supplements a detailed re- quest. then generates a corresponding sounding video.
Que: Create video where person crumples paper and
MultiTurn
Ans: Do you want the sound of paper being crumpled up can he heard as ...! Que: Yes!
Und2Cien
Model first answers an audio-video- related question, and the generate/extend corresponding sounding video.
Que: What emotion is the crowd ex- pressing? Ans: Surprise and mixture of applause. Que: Show me video of crowd with silence and sacredness.
FoleyCrafter(9a to generate corresponding audios for both the pre-edit and post-edit video clips, thus enriching training scenarios for AV-based editing tasks.
Multi-turn Conversation: We carefully design three representative and practical forms of multi-turn generation: proactive generation, rethink generation, and petersstandiggto-gneereee In proactive generation. the model is expected identify user preferences from prior dialogue and incorporate them into the generated sounding video. In rethink, the model receives an ambiguous initial instruction and proactively suggests more complete and specific generation prompt.
(Und2Gen).
To construct such data, we reuse sounding videos from TAVGBench as generation targets and employ GPT-4o to synthesize dialogues based on their corresponding captions. Specifically: (1) for proactive scenarios, we remove one key entity or concept from the caption and create a preceding dialogue turn that embeds a preference, simulating preference-aware generation (see prompt template in Fig. A2 (2) for rethink, we generate a highly abstract version of the caption as the user's vague instruction. Then, the original detailed caption is reformatted using instruction generation templates to serve as the model's suggested clarification.
As for Und2Gen, the model first receives sounding video and engages in one round of understanding dialogue (from Sec. C.2), followed by a generation task such as producing a new audio-vides clip performing AV-extension based on prior understanding. Inspired by NEXT-GPT [8I, we BERTScore [92 to identify semantically similar caption pairs in TAVGBench, and instruct GPT-4o construct conditional generation prompts accordingly.
In total, JavisInst-Gen contains 90K high-quality QA samples (see detailed statistics Fig. 5). supporting wide range of real-world generation scenarios.

---

Generate then Understand
Understand then Generate
Presen: the visuals with sudio track. A wearing black slist hullding slick his hand, accompariod by Enotier man standing next to him the background, woman is singing along to the music playing.
What is the overall theme the video?
Sure! I will incorpecate the visuals with audio tack featuring the black shirt with stick, alongside the bockground.
other while the
Geographical education. The video shows in acrial view of the Forth with misic, suggasing a facus on geographical oduciiiion.
Keep the visual style intact and extend its timeline.
Yap, I will maintain the visual sivie while excending the tmeline, incoeporating the besutiful aetial view Earh with the music enhancing the amooppere.
the woman singing the audio also visible in the video?
No. The women singing un the backgroundd
How many people be sdentifldd the video and audio?
Throe, Two in the video and one in the andio.
Proactive Generation
Rethinking Generation
Hi, JavivGPT When do you ahout sound drums
Hi, InvieIPPT Create video of man riding a horse down gravel path, followed by jingling goats
tlink they produce an amazing and dytanti sound!
Do you want to tum this into video? The sound of foodsteps can be heard on gravel path as bards chirp the backgnound. A man seen riding home down dirt road, with lesd of goans falkowing belind him. The landscape appears rugged and rural, with the man leading the atmuls in seomingly caim and lesurely manner. The goses move steadilyy along, their bells jingling as they trot along the path.
Develop a video to accunnely ly represent these points, man is energtically performing on stage, pusitionaal front of microphome, indiaating that may also he singing during the perfanncce shating lim, highlightingg movemments passionately performs, creating an engaging simoephore.
stage lights are
video captures the energetic periormance of sua playing on stage, passion
Roger Blat.
stmoqphare with
Grest idea! will manssoon this video sound, cepturing stmophar
stage highlighting
Figure A3: Examples of multi-turn comprehension and generation from JavisInst-Omni.
D Architecture of Joint Comprehension and Generation of Sounding Videos
D.I Design Choices of Generative MILM
Designing a unified architecture for both comprehension and generation tasks remains a central challenge. As there no previous exploration in JAV domain (we are the first one), mainly summarize and compare three representative paradigms in Fig. A4 under the text-image context:
LLM+DIT. Our JavisGPT adopts modular yet effective LLM + DiT architecture, which has been widely verified in recent multimodal works [8]]5]] for its balance between efficiency and generation quality. This architecture separates semantic understanding (handled by the LLM) from generative decoding (handled by a diffusion transformer), allowing task specialization and easier optimization. Moreover, the use of learnable queries and a conditioned DiT enables flexible generation from various modalities with strong temporal coherence.
attempt to unify textual autoregression and multimodal diffusion within single large language model. However, these MonoL.MMs suffer from severe optimization difficulties due to conflicting training objectives and heterogeneous output formats 99. The lack of architectural decoupling often leads to poor convergence and unstable generation, especially for high- -igh--iimnsional outputs like audio or video.
MonoLLMs. Recent works 996 84 45
25

---

Diffusion Model
Img. Denoise
Imy: Decolar
Large Language Model
Large language Model
Large Language Model
Emg: Ensrder
Emg. Ensasler
Img Enender
(A) LLM+DIT
(B) MonoLLM
(C) LLM+
Figure A4: Choices of unified architecture for joint understanding and generation. (A) Our JavisGPT adopts the widely verified LLM+DiT architecture [1 for efficiency and efficacy. (B) that integrate textual auto-regression and multimodal diffusion into single LLM, causing severe optimization difficulty. (C) The LLM+AR+
MonoLLMs 196 84
paradigm [77 82 brings
extra training efforts encoder-decoder modules for JAVG and high inference cost to predict numerous multimodal tokens in the auto rugo-ssegession behavior.
VIDII
IVERO
VIDai
SAANG]
[+]
LLM
LLM
LLM
[S]
[SNC]]
[+]]
DIT
DiT
DiT
(A) NEXT-GPT
(B) MetaQuery
(C) JavisGPT
Figure A5: Choices for MILMM DiT combination. (A) NEXT-GPT 81 suffers from degraded generation quality and diversity due its forced regression of all generation instructions to fixed set of special tokens (i.e., <VIDI>, <VID2>, ... <VIDn>). (B) MetaQuery [51] removes the support of text encoder, which leads to training instability. (C) Our JavisGPT combines learnable query embeddings with supportive text encoder, enabling stable training and the generation of high-quality and diverse audio-video content.
The autoregressive incoder-decodee paradigm 77 82 introduces addi- tional training complexity, which requires maintaining a heavy encoder-decoder pipeline to bridge multimodal embeddings with autoregressive prediction on discrete [82 or continuous [75] token embeddings and trying to avoid information loss. Besides, this design also incurs significant inference costs, as it must sequentially predict large volumes of multimodal tokens. For instance, when using 512 tokens per frame, generating one-minute video clip (even without audio) at 24 fps would require generating 512 24 x 60 237,280 tokens, making it less practical for real-time high-resolution generation.
ALM+AR+Decoun
In conclusion, the LLM+DiT paradigm is the best option for current JAVG field, which has also been verified by our comprehensive experiments in Sec. 6 However, the LLM+AR+Decoder architecture may show potentials further bridge the unified task of comprehension and generation. The key insight is simple and straightforward: after unifying understanding and generation AV representations into the same space, the generation embeddings can be also "seen" by comprehension tasks. As discussed in Sec. [ this feature can mutually strengthen both comprehension and generation capabilities after scaling up model capacity and data quantity. We hope this can bring more inspiration in the community.
D.2 Design Choices of LLM-DiT Combination
In Fig. we also compare several designs of L.L.+DDTT combination, which further highlights the advance of our JavisGPT's architecture in performance, diversity, and training stability.
26

---

NEXT-GPT. NEXT-GPT 88 formulates generation as a classification task over fixed vocabulary of video IDs (e.g., <VIDI>, <VID2>, <VIDn>>, ..VDD>,....DDD>>, forcing all instruction prompts to regress this small set of special tokens. This rigid formulation limits the expressiveness of conditional instructions and severely constrains the diversity and fidelity of generated audio- outputs, which is consistent to NEXT-GPT's poor generation performance in Sec. [6.1 Additionally, it restricts generalization to novel or fine-gr tine-grained instructions outside the pre-defined token set.
MetaQuery. MetaQuery [ST] introduces a learnable query mechanism but removes the auxiliary sup- port of the text encoder of DiT, relying solely on query tokens to interface with the generation model. This approach often leads to training instability, especially when scaling to complex multimodal data like audio-video. The absence of semantically grounded text features weakens the model's ability to align instructions with generated content.
JavisGPT. Our JavisGPT integrates the advantages of both sides by combining learnable query embeddings with the semantic guidance of a pretrained text encoder. The text encoder offers strong grounding for conditional semanties, while the queries serve as adaptive, trainable intermediaries to control the generative diffusion model. This hybrid design results in stable training dynamics and supports the generation of high-quality, diverse, and instruction-aligned sounding videos. Moreover, it enables better generalization to a wide range of instructions, including free-form, multi-turn, or mm-conditional prompts.
E Supplementary Investigation on JavisGPT
E.
Details for Human Evaluation on Interleaved Conversation
In Sec. 6.1 we present comparative evaluation with UnifiedIO-2 42 and NEXT-GPT 81 in interleaved comprehension and generation scenarios. Since no existing benchmark is available for this setting, we conduct human evaluation, as detailed below:
Data Collection. We use GPT-4o to construct 100 multi-turn dialogue samples based on the four scenarios illustrated in Fig. A3 including Gen2Und, Und2Gen, Proactive, and Rethink (25 samples each).
Evaluation Criteria. Three unbiased human annotators independently rated the model outputs scale of 0-5 across five dimensions:
1. Instruction Following:
whether the model follows the user instruction to generate coherent textual response and a matching audio-video output;
2. Question Answering: whether the model accurately understands the input audio-video content and provides correct answers;
3. Generation Quality: the overall quality, coherence, and synchrony of the generated audio-video content;
4. Context Reasoning: whether the model can integrate user preferences implied in the dialogue history into its response or generation;
5. Proactive Thinking: whether the model recognize vague under-specified instructions and proactively ask clarifying questions or complete the intent.
Result Analysis. The evaluation results are shown in Fig. from which we draw several key conclusions:
NEXT-GPT shows poor instructionnfollloiing ability and often fails to generate audio-video outputs-only 33 out of 100 cases succeeded. This can be attributed insufficient fine-tuning (only 5K samples used).
Both NEXT-GPT and Unified10-2 suffer from low generation quality. As discussed in Sec. D.2 NEXT-GPT's reliance on fixed special tokens for all instructions limits both generation diversity and fidelity. UnifiedIO-2. lacking a dedicated DiT-based generator, resorts to generating keyframes and stitching them into video, resulting in subpar temporal coherence and realism.
JavisGPT significantly outperforms both baselines in question-answerrin benefiting from the scale and diversity of the JavisInst-Omnl dataset.
Due to the lack of training on preference reasoning and proactive dialogue, both NEXT-GPT and Unified1O-2 exhibit weak context reasoning and proactive generation, whereas JavisGPT con-
27

---

sistently shows stronger contextual understanding and responsiveness, underscoring its practical utility and superior alignment with real-world conversational needs.
The results further highlight the strong capability of our JavisGPT.
E.2 Investigation on Backbone LLM
In the manuscript, we mainly take the Qwen2.5- VL [5] as the backbone LLM to build our Jav- isGPT. Here provide further investigation on the base LLM with a slightly weaker Qwen2-
VL [76] model. After the same training data and procedure, we compare the performance variation in Fig. A6]for further analysis. Accord- ingly, upgrading the language backbone from Qwen2-VL to Qwen2.5-VL consistently im- proves performance across all evaluation tracks, including unimodal understanding (V-Und, A- Und), audio-visual comprehension (AV-Und), and audio- visual generation (AV-Gen). Notably, the gain is most pronounced in the generation task (AV-Gen). where Qwen2.5-Ve outperforms Qwen2- VL by substantial margin. These results highlight that a more capable language foundation model can significantly benefit not only text-based reasoning but also cross-moda alignment and response generation. The consistent improvements across both understanding and generation tasks validate the critical role of the LLM backbone in scaling multimodal capabilities of our JavisGPT.
A-Aad
AV43md
AV Gen
Qwenž Qwend: 5-VL
Evaluation Performance
Figure A6: Ablation on Backbone LLMs.
Qualitative Case Studies of JavisGPT
More understanding and generation cases can be found in Fig. A7
28

---

Audio Comprehension
Generate then Understand
Is the sound cansedl by presure Freemagsiine wood?
Make quick clip with soundtrack. The machine can be heard numning steadily in the background a close up shot of car's engine with a light shining showing denriled components and mechunical parts.
Yes. It is trosed by pressure agpinst wood.
Sure! together short that captures machine, painel with close-up the engine showesse inniecte componictts.
Video Comprehension
What happened after the took the food?
A. Ale the medicine. B. Tidied the blanket.
Put down the cuprglass bottle. D. Took the box.
Does the machine producing stendy sound uppear video?
The engine that produces the steady sound appears the videa.
A. Ate the medicine.
When is the machine heard running steadily?
Audio-Video Comprehension
Belore up
car's eagine
During the shot the engine when light turned off
How instuments the video did sound from beginning to and? Ansover the using simple word
B. Dusing the close-up shot of the cat's engine.
Proactive Generation
like acttings
Zero:
Wow, dark settings really have such string vibe!
Audio-Video Generation
Combine nurration and rendition this large crowe gathored in setting, raising their hunds the The muficnce appears engagal and escitol, with various individuals visible within the crowe. The lively and energetic atmosplizee, In the audio, woman be heard singing and whouping playing the background
forn complete video goers
Make sounding-tiloo out of this As serial view of bench with white aind and shore: Tunquose waters waths waves emsching onto the shore can seen, four binds flying overhead. Severai
umbrallas lining the
with music
the banch. The sound of wind blowing and unter splishing he harnd throughot..
Sure! I'll blend the narzaion with visuals to bring the concert sorne life and capture its vibrani enengy dark setting.
Great idea! I will create a soundingvides with the beautiful setial view of the beach with all the described elements.
Figure A7: More qualitative results for joint understanding and generation.
29

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction accurately reflect the contributions and scope of the paper. We ensure that the stated contributions are directly supported by our method, experiments, and analysis presented in the main text.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contribu- tions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals motivation as long as it is clear that these goals not attained by the paper.
Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of the work are comprchensively discussed in Sec.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to vi- olations these assumptions (e.g., independence assumptions, noiseless settings, model well-specifficttion asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or a speech-to opeech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by review- grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
30

---

Justification:
This paper does not introduce new assumptions or proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof's in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experi- mental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
The paper provides all necessary implementation details required to reproduce the main experimental results, including dataset descriptions, model architectures, training configura- tions, and evaluation protocols in Sec. [ and Sec. [B] Additionally, we include an anonymous code repository for reproducibility: https:/ hatps:///anonnpnn annnymous 4open. science/r/Javisst
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. general. releasing code and data is often one good way to accomplish this, but reproducibility results, access to a hosted model (e.g., in the case a large language model), releasing of a model checkpoint, other means that are appropriate to the research performed.
also be provided via detailed instructions for how to replicate the
While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution be a way to access this model for reproducing the results or way reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
new model (e.g., a large language model), then there should either
(d) We recognize that reproducibility may be tricky some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of fosed-source models, may that access to model limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
31

---

Answer: [Yes]
Justification: We provide open access to both the code and data via an anonymous repository at https://anonymours www.ee...ooo.ooo scwience/r//aiii//
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: htttpp::/niiss httpp//////is.ccppbllii.
guides/CodeSubee ssioooolicii for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines futtps://nip httpe//gidee/e/SSuuiee oslicillicyyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
The paper provides all necessary implementation details required to reproduce the main experimental results, including dataset descriptions, model architectures, training configura- tions, and evaluation protocols in Sec. and Sec. [ Additionally, we include an anonymous code repository for reproducibility: https: httts::/annoyymouut 4open. science/r/lavisGFT
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We follow previous works fix the random seed as 42 for fair comparison and repro- ducibility, The results in Sec. show large improvement against baseline models, emphasizing the statistical significance.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
32

---

The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of eπrоrs is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). If crror bars are reported in tables or plots, The authors should explain in the text how they calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experi- ments?
Answer: [Yes]
Justification: We report the type of compute resources used for all experiments in Sec. and Sec. B to facilitate reproducibility and resource estimation.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experi- mental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We have carefully study the NeurlPS Code of Ethics and conforms to it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed the positive and negative societal impact the proposed work in Sec. A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimesss considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We recognize the potential risks associated with releasing model and data and discuss in A
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer:
[Yes]
Justification: We conforms to the terms of service and license policy of existing datasets in their official websites.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswil thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that re- ee-aakaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
34

---

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We introduces new assets, including a benchmark, pretrained models, evaluation metrics, which accompanied by comprehensive documentation at https: ///onnnoououuu
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/coddemmodel part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects, We only perform human evaluation on three volunteers.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to NeurIPS Code Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. We only perform human evaluation on three volunteers.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
35

---

Question: Does the paper describe the usage of LLMS if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration not required.
Answer: [NA]
Justification: LLMs were used only for language editing and formatting assistance.
Guidelines:
The answer NA means that the core method development in this research does not involve
LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https: / for what should or should not be described.
36


Reviews:
[{"review_id": "CPS94M8beo", "reviewer": "Reviewer_MnnK", "summary": "# Summary\n\nThis paper proposes a novel model for unified understanding and generation of synchronized audio-video. The model is built upon the Qwen2.5-VL-7B-Instruct backbone. For encoding, images are processed using Qwen2.5-VL’s visual encoder, while audio is encoded with BEATs. An additional SyncFusion module, based on cross-attention, is introduced to obtain synchronized audio-video features, with its output dimensionality aligned to that of the encoded video features. These fused features are then fed into the LLM backbone. For generation, multiple query tokens are provided to the LLM backbone, which then produces latent conditions for an external DiT module responsible for synchronized audio-video generation. The output conditions are aligned with the T5-XXL text encoder, and the DiT module remains frozen during training. The authors further propose a hierarchical query token approach, where the output conditions of these additional queries are aligned with the Spatial-Temporal Prior encoder of JavisDiT. The final loss function comprises four terms: (1) semantic alignment loss, (2) ST-prior alignment loss, (3) denoising objective (not used during pre-training), and (4) NTP objective. The model is trained in three stages: (1) a pre-training stage to align audio inputs and multimodal outputs, (2) an audio-video fine-tuning stage to enhance synchronized audio-video comprehension and generation, and (3) instruction-tuning to improve instruction-following and reasoning abilities. Additionally, the authors introduce the JavisInst-Omni dataset.", "questions": "1. **Lines 217–220**: This experimental finding is quite interesting. I am very curious about it. Generally, for video generation models, a two-stage approach involving prompt engineering (PE) and DiT is used: the PE stage leverages a language model or MLLM to generate a caption for the target video, and then DiT uses a text encoder to obtain the conditions for video generation. Although some recent methods have attempted to bypass the PE stage by directly generating such conditions with an LLM, none have shown superior video generation performance compared to the traditional PE+DiT approach. Therefore, the finding in this paper that JavisGPT outperforms JavisDiT is particularly intriguing. Can you provide more insights into this? What are the key factors that allow the latter solution to surpass the PE+DiT approach? How do you ensure that, even with only a vague textual condition (and not a clear target video caption), the performance of direct condition generation can exceed that of PE+DiT? Or what are the reasons for which the previous non-PE approaches fail?\n\n2. **Lines 109–114**: The authors freeze JavisDiT for the sake of training efficiency and simplicity. What would happen if JavisDiT were also trained (i.e., unfrozen)? Could this potentially further improve performance? What are your thoughts on the possible training stability issues when unfreezing JavisDiT, especially in terms of avoiding significant degradation in DiT performance? (I know that JavisDiT is frozen in this work, but I am very curious about this.)\n\n3. When will this model be open-sourced?\n\nThis paper is impressive. I am willing to increase my score if the weaknesses and my questions are addressed.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 4, "strengths_and_weaknesses": "# Strengths\n\n1. This work represents a significant milestone in the field of sounding video understanding and generation.\n2. The technical contributions are substantial and introduce meaningful innovations.\n3. The experiments are well-designed, and the findings are both solid and interesting.\n4. JavisGPT demonstrates impressive performance.\n5. The ablation studies are thorough and provide valuable insights.\n\n# Weaknesses\n\n1. Why did the authors choose BEATs (2022) as the audio encoder, instead of adopting more recent advancements such as WavTokenizer (2024)[1]?\n2. It would be beneficial to provide more details about the evaluation metrics for audio-video generation, perhaps in the appendix.\n3. What are the benefits of cross-attention mechanism (SyncFusion) compared with interleaved concatenation ([Vclip0, Aclip0, Vclip1, Aclip1, ...]) for audio-video synchrony?\n\n[1] wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling", "was_revised": true, "final_justification": "The authors have addressed most of my concerns and questions.", "rebuttal": "We sincerely thank the reviewer for the recognition and valuable insights, which have also been inspiring to us. We hope the following responses will help address the remaining concerns.\n\n> **W1**: Why did the authors choose BEATs (2022) as the audio encoder, instead of adopting more recent advancements such as WavTokenizer (2024)?\n\n**Ans**: There are several main reasons:\n1. We follow prior works (e.g., Video-Salmonn[1], LongVALE-LLM[2]) in using BEATs, due to its strong generalization ability in audio representation.\n2. BEATs is optimized for sound effects, which aligns well with the focus of our work. Differently, WavTokenizer is primarily designed for speech processing (although it also supports music and audio), which is not well aligned with the focus of our study.\n\nIn future, we are happy to explore WavTokenizer for speech understanding or generation in particular.\n\n> **W2**: It would be beneficial to provide more details about the evaluation metrics for audio-video generation, perhaps in the appendix.\n\n**Ans**: Thank you for the suggestion. We have provided a brief explanation of the metrics used to evaluate audio-video quality, consistency, and synchrony in the supplementary material (appendix.pdf, Section B.3), including their definitions and reference computation methods. We will include more detailed descriptions of the calculation procedures in the revised paper to improve readability.\n\n> **W3**: What are the benefits of cross-attention mechanism (SyncFusion) compared with interleaved concatenation ([Vclip0, Aclip0, Vclip1, Aclip1, ...]) for audio-video synchrony?\n\n**Ans**: SyncFusion offers both performance and efficiency benefits, as supported by quantitative results below (partially copied from Table 4 in the paper):\n- In terms of performance, SyncFusion injects audio information directly into visual frames, which reduces the burden on the model to capture synchrony across separate audio and video tokens, thereby improving comprehension and accuracy.\n- In terms of efficiency, SyncFusion reduces the number of tokens to process and avoids interleaving audio and video tokens, significantly reducing memory overhead and lowering inference latency.\n\n|   Mechanism   | AVQA Acc ↑  | MU-AVQA Acc ↑ | AVSD Acc ↑ | Inference Latency ↓ |\n|:----------:|:--------:|:------------:|:--------:|:-----------------:|\n| Interleave |   93.3   |     80.6     |   61.6   |        555ms       |\n| **SyncFusion** | **93.4** |   **81.4**   | **62.0** |    **224ms**     |\n\n> **Q1**: Lines 217–220: The finding that JavisGPT outperforms JavisDiT is particularly intriguing. Can you provide more insights into this? What are the key factors that allow the latter solution to surpass the PE+DiT approach? How do you ensure that, even with only a vague textual condition (and not a clear target video caption), the performance of direct condition generation can exceed that of PE+DiT? Or what are the reasons for which the previous non-PE approaches fail?\n\n**Ans**: \nFirst, we would like to clarify that in the experimental setting of Lines 217–220 (Table 3), both JavisDiT and JavisGPT receive the same detailed textual input, without involving additional prompt engineering process. Compared to the standard text encoder (T5) used in DiT, JavisGPT leverages an generative LLM (Qwen) to integrate information, which offers stronger semantic understanding and reasoning capabilities. For example, given the instruction \"a cat jumps onto a table and knocks over a cup\", a conventional encoder tends to process tokens like “cat”, “table”, and “cup” as discrete elements. In comparison, the LLM captures the causal and temporal relationship between actions, inferring that jumping onto the table leads to knocking over the cup. This supports the generation of more coherent and logically structured videos. This design philosophy aligns with recent works like Hunyuan-Video[3], where LLMs are also used as semantic extractors to guide generation.\n\nRegarding vague textual conditions, traditional PE+DiT pipelines rely on an external LLM to expand user input into a detailed caption. However, the style and distribution of such generateds caption may differ from the text that DiT has seen during training, including phrasing, sentence structure, and detail granularity. This creates a style gap that hinders generation quality. Differently, JavisGPT jointly trains the LLM and DiT modules, effectively reducing this mismatch. During inference, JavisGPT proactively completes the vague instruction into a richer caption, and upon user confirmation, passes the full semantic context to the DiT decoder via JavisQuery for sounding video generation. This mechanism is illustrated in Appendix Figure A3 (Rethinking Generation), which shows how our framework handles vague instructions while maintaining semantic fidelity.\n\n> **Q2**: Lines 109–114: The authors freeze JavisDiT for the sake of training efficiency and simplicity. What would happen if JavisDiT were also trained (i.e., unfrozen)? Could this potentially further improve performance? What are your thoughts on the possible training stability issues when unfreezing JavisDiT, especially in terms of avoiding significant degradation in DiT performance? (I know that JavisDiT is frozen in this work, but I am very curious about this.)\n\n**Ans**: Great question. We would draw your attention to the observation in MetaQuery[4], where unfreezing DiT and training it jointly with other components could theoretically lead to better performance. However, there are several practical challenges that must be addressed:\n\n1.  **Training stability** is the primary concern. In fact, even with the DiT frozen, if we skip the MM-PT (Multimodal Pretraining) stage and directly connect the modules using diffusion loss, the system tends to completely break, rather than merely suffering from “significant degradation” in DiT performance. Therefore, if we were to train JavisDiT jointly, we would need to adopt a multi-stage training strategy or apply advanced stabilization techniques such as learning rate scheduling and gradient clipping.\n\n2. Another key constraint is **memory usage**. JavisDiT has over 3 billion parameters, and unfreezing it would add approximately 48GB (16 × 3GB) of GPU memory consumption. This would require higher-level ZeRO parallelism, parameter offloading, and other advanced memory optimization strategies, which would significantly increase the engineering complexity of training.\n\nThat said, we believe that with extensive optimization, jointly training JavisDiT could indeed lead to further performance gains, and we plan to explore this direction in future work.\n\n> **Q3**: When will this model be open-sourced?\n\n**Ans**: We are currently organizing all the code, data, and model checkpoints, and plan to open-source the project as soon as possible.\n\n---\n\n[1] Sun G, Yu W, Tang C, et al. video-SALMONN: speech-enhanced audio-visual large language models[C]//Proceedings of the 41st International Conference on Machine Learning. 2024: 47198-47217.\n\n[2] Geng T, Zhang J, Wang Q, et al. Longvale: Vision-audio-language-event benchmark towards time-aware omni-modal perception of long videos[C]//Proceedings of the Computer Vision and Pattern Recognition Conference. 2025: 18959-18969.\n\n[3] Kong W, Tian Q, Zhang Z, et al. Hunyuanvideo: A systematic framework for large video generative models[J]. arXiv preprint arXiv:2412.03603, 2024.\n\n[4] Pan X, Shukla S N, Singh A, et al. Transfer between modalities with metaqueries[J]. arXiv preprint arXiv:2504.06256, 2025."}, {"review_id": "NnQtzl13rL", "reviewer": "Reviewer_SwFQ", "summary": "This article introduces JavisGPT, which is a unified MLLM specifically designed for audio-video coherent understanding and generation. It adopts an encoder-LLM-decoder architecture based on Qwen2.5-VL, BEAT, and pre-trained JAV-DiT generator, and bridges it through the SyncFusion module. Meanwhile, the paper constructs and uses JavisInst-Omni, which is an internal dataset consisting of 200K GPT-4o audio-video-text dialogues. In the understanding and generation benchmark tests of the paper, especially in the time synchronization task, JavisGPT's performance is significantly superior to that of previous MLLMs.", "questions": "- In addition to the concerns raised, the performance gains from SyncFusion appear relatively modest in table 4, and the reduced latency seems heavily tied to using fewer encoded AV tokens. This doesn’t convincingly show its superiority over Bidirectional Cross-Attention. \n- Could the authors provide additional experiments or insights to clarify where SyncFusion outperforms Bidirectional Cross-Attention, both in accuracy and efficiency?\n- Furthermore, I’m curious about how compressing audio-visual tokens impacts speed and other metrics. specifically:\nDoes cutting down on AV tokens significantly boost inference speed?\nAre there hidden costs, like a drop in accuracy or other performance trade-offs?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength \n- This study introduces a new instruction dataset, which covers a wide range of real-world scenarios.\n- SyncFusion effectively aligns fine‑grained audio and video features, enabling coherent sounding‑video generation from natural instructions.\n- The three-stage training pipeline is well-conceived, offering a systematic way to incrementally build and refine the model's complex multimodal capabilities.\n\nWeakness\n- The human evaluation in this study involved only three volunteers, which may introduce bias due to the small sample size and limited diversity of assessors. Future work could benefit from a larger and more diverse group of evaluators to enhance the reliability and generalizability of the findings.\n- The paper introduces heavy components like JAV-DiT and SyncFusion, which increase computational complexity and may hinder real-time or resource-constrained deployment. Though some comparisons with alternative methods are included, the scope is limited and does not cover simpler options like mlp. Thus, it remains unclear if the performance gains justify the higher computational costs and latency.", "was_revised": true, "final_justification": "I thank the authors for their comprehensive and timely rebuttal, which addressed my main concerns. The expansion of the human evaluation pool and the additional analysis on SyncFusion’s efficiency and accuracy provide valuable clarification. The updated results and further contextualization of the trade-offs strengthen the overall case for the paper. While there is still room for improvement in the scale of human evaluation and breadth of baseline comparisons, the technical merits and thoughtful design of the framework outweigh these limitations.\n\nGiven the clarified contributions and the added experimental evidence, I believe the paper makes a meaningful and relevant contribution to the field. I am maintaining my rating of borderline accept.", "rebuttal": "> **W1**: The human evaluation in this study involved only three volunteers, which may introduce bias due to the small sample size and limited diversity of assessors. Future work could benefit from a larger and more diverse group of evaluators to enhance the reliability and generalizability of the findings.\n\n**Ans**: Thank you for the suggestion. During the rebuttal period, we added two additional volunteers with different backgrounds to the human evaluation. The updated averaged results are reported below, and the original conclusion holds, where JavisGPT consistently outperforms NExT-GPT and UnifiedIO-2 in both instruction-following and audio-visual understanding/generation capabilities. In future work, we plan to expand the evaluation along more dimensions and involve a more diverse group of assessors to further improve the robustness and generalizability of the human evaluation.\n\n| Model       | Instruction Following | Question Answering | Generation Quality | Context Reasoning | Proactive Thinking |\n|-------------|:---------------------:|:------------------:|:------------------:|:-----------------:|:------------------:|\n| UnifiedIO-2 |     74     |     71    |     58    |     63     |     52     |\n| NExT-GPT    |     52     |     74    |     44    |     69     |     59     |\n| JavisGPT    |     **94**     |     **90**    |     **88**    |     **80**     |     **81**     |\n\n\n> **W2**: The paper introduces heavy components like JAV-DiT and SyncFusion, which increase computational complexity and may hinder real-time or resource-constrained deployment. Though some comparisons with alternative methods are included, the scope is limited and does not cover simpler options like mlp. Thus, it remains unclear if the performance gains justify the higher computational costs and latency.\n\n**Ans**: Thanks for the comment. As discussed in Section 4, our core contribution lies in the overall framework design and optimization, where most components are modular and configurable to suit different deployment scenarios, including the sizes of the LLM and DiT backbones that dominate computational cost. The additional adaptors have limited impact on overall computation. For example, SyncFusion (CrossAttn+MLP) and standalone MLP layers introduce only 154.2M and 102.8M parameters respectively, accounting for just 1–2% of a 7B-scale LLM.\n\nIn fact, the “Concat” baseline in Table 4 is functionally equivalent to an MLP-style adaptor, as any mapping from AV representations to the LLM input space requires some form of lightweight projection. The results show that SyncFusion achieves better performance with comparable or even lower computational overhead, justifying its inclusion in the system.\n\n\n|   Component   | AVQA Acc ↑  | MU-AVQA Acc ↑ | AVSD Acc ↑ | Param. Num. | Inference Latency ↓ |\n|:----------:|:--------:|:------------:|:--------:|:--------:|:-----------------:|\n| Concat (MLP) |   93.3   |     80.7     |   61.3   |   **102.8M** |     246ms       |\n| **SyncFusion** | **93.4** |   **81.4**   | **62.0** |  154.2M | **224ms**     |\n\n> **Q1**: In addition to the concerns raised, the performance gains from SyncFusion appear relatively modest in table 4, and the reduced latency seems heavily tied to using fewer encoded AV tokens. This doesn’t convincingly show its superiority over Bidirectional Cross-Attention.\n\n**Ans**: SyncFusion is designed to benefit both performance and efficiency:\n\n- Efficacy: The performance gains are meaningful. As shown in Table 4, replacing Interleaved concatenation with Bi-directional Cross-Attention actually leads to a drop in performance on AVQA and AVSD tasks. It suggests that simply adding more attention computation does not guarantee improvement, whereas SyncFusion brings gains by performing frame-level injection and alignment, which reflects its task-specific design and usefulness.\n\n- Efficiency: In the LLM era, latency is tightly coupled with token count. Given that the computational backbone (LLM and DiT) remains unchanged, reducing the number of AV tokens directly lowers the inference cost during the prefill stage during deployment.\n\n|   Component   | AVQA Acc ↑  | MU-AVQA Acc ↑ | AVSD Acc ↑ | Inference Latency ↓ |\n|:----------:|:--------:|:------------:|:--------:|:--------:|\n| Interleave |   93.3   |     80.6     |   61.6   |  555ms |\n| BiCrossAttn |   93.2   |     80.7     |   61.1   |  263ms |\n| **SyncFusion** | **93.4** |   **81.4**   | **62.0** | **224ms** |\n\n> **Q2**: Could the authors provide additional experiments or insights to clarify where SyncFusion outperforms Bidirectional Cross-Attention, both in accuracy and efficiency?\n\n**Ans**: The main limitation of Bidirectional Cross-Attention lies in **information redundancy**. Based on the results in Table 4, we can draw the following insights:\n\n- Accuracy: After mutual fusion, video tokens carry auditory information and audio tokens also carry visual information, making it harder for the model to distinguish which modality to rely on for specific questions. This increases the complexity of reasoning and leads to reduced answer accuracy.\n\n- Efficiency: Compared to SyncFusion, Bidirectional Cross-Attention retains a large number of enhanced audio tokens, which adds to the computational load and increases latency during inference.\n\nIn summary, SyncFusion removes redundant audio tokens while preserving essential cross-modal alignment, thereby improving both accuracy and efficiency over Bidirectional Cross-Attention.\n\n> **Q3**: Furthermore, I’m curious about how compressing audio-visual tokens impacts speed and other metrics. specifically: Does cutting down on AV tokens significantly boost inference speed? Are there hidden costs, like a drop in accuracy or other performance trade-offs?\n\n**Ans**: Naively compressing audio-visual tokens can improve inference speed to some extent, but it often comes at a significant cost in performance. As shown in the table below, during rebuttal we conducted a simple experiment where AV tokens were compressed via 2× average pooling, and observed the following:\n\n- Inference speed showed only a minor improvement, where latency dropped from 246 ms to 206 ms. As discussed earlier, since AV token count only affects the prefill stage, the speedup is limited and not particularly significant.\n\n- Performance dropped noticeably, where AVSD accuracy particularly decreased by nearly 3%, indicating that naive compression leads to substantial information loss.\n\nThese results are consistent with our original motivation: instead of compressing AV tokens directly, our SyncFusion strategy reduces token redundancy while preserving critical cross-modal information, achieving both speed and accuracy gains.\n\n|   Compression Strategy   | AVQA Acc ↑  | MU-AVQA Acc ↑ | AVSD Acc ↑ | AV Tokens | Inference Latency ↓ |\n|:----------:|:--------:|:------------:|:--------:|:--------:|:-----------------:|\n| None  |   93.3   |     80.7     |   61.3   |    3.5K    |     246ms       |\n| AvgPooling  |   91.9   |     78.4     |   58.6   |    **1.8K**    |    **206ms**       |\n| **SyncFusion** | **93.4** |   **81.4**   | **62.0** |  2.0K  |     224ms     |"}, {"review_id": "Svnz3v2KPe", "reviewer": "Reviewer_3pH1", "summary": "The paper present JavisGPT, an encoder‑LLM‑decoder system that unifies synchronized audio‑video (AV) comprehension and generation. Key ingredients include:\n\n(1) SyncFusion: a cross‑attention module that merges frame‑aligned audio features into video patches, yielding SyncAV tokens for fine‑grained spatio‑temporal reasoning.\n\n(2) Hierarchical queries (JavisQuery / JavisCond / ST‑prior) that map LLM hidden states to a frozen JAV‑DiT diffusion generator, enabling one‑shot, temporally coherent AV synthesis. \n\n(3) A three‑stage training pipeline—multimodal pre‑train, AV fine‑tune, and instruction tuning—driven by the 200 k‑dialogue JavisInst‑Omni dataset .", "questions": "1. How does SyncFusion differ fundamentally from earlier audio–video fusion layers such as the Q-Former/FiLM approaches used in AVFormer and NExT-GPT? Where is the genuine innovation beyond frame-aligned cross-attention?\n\n2. If the temporal-spatial prior query is removed—keeping only JavisQuery and JavisCond—how much do synchrony and generation quality drop? Please provide an ablation isolating its contribution.\n\n3. Could the frozen DiT be replaced with a lighter autoregressive or VQ-GAN decoder without retraining the LLM adapters? Any preliminary experiments or rationale?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n(1) Covers eight comprehension benchmarks, text‑to‑AV generation metrics, and a human study on interleaved dialogues; baseline list is current and competitive.\n\n(2) Build an model that simultaneously understands and generates fully synchronized video and audio with a shared backbone.\n\n\n**Weaknesses**\n\n(1) Limited conceptual novelty. Most of JavisGPT’s building blocks already exist in the literature and the paper does not clearly articulate what is fundamentally new beyond their specific combination:  SyncFusion is essentially a Q‑Former/FiLM‑style cross‑attention layer applied to frame‑aligned audio features; similar audio‑video fusion schemes appear in AVFormer [1]. Hierarchical queries (JavisQuery / JavisCond / ST‑prior) resemble the two‑level adapter hierarchy of Control‑Net and Uni‑Adapter, while the frozen DiT decoder mirrors Make‑A‑Video and VideoComposer. The three‑stage training recipe (multimodal pre‑train → task fine‑tune → instruction tuning) is standard practice inherited from LLaVA, MiniGPT‑4, and NExT‑GPT.  \n\n(2) The new 200 k “JavisInst‑Omni” dialogues are synthetically generated from GPT‑4 with task‑specific prompts, following LLaVA‑v1.5 and NExT‑GPT‑Lite, rather than collected from humans.\n\n[1] Seo P H, Nagrani A, Schmid C. Avformer: Injecting vision into frozen speech models for zero-shot av-asr[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 22922-22931.", "was_revised": true, "final_justification": "My concerns has been solved. Although the novelty of this paper is limited, I think this is a good paper.", "rebuttal": "> **W1**:  Limited conceptual novelty. Most of JavisGPT’s building blocks already exist in the literature and the paper does not clearly articulate what is fundamentally new beyond their specific combination.\n\n**Ans**: We appreciate the reviewer’s concern regarding conceptual novelty. As discussed in Sec.4 Contribution Statement in this paper, we agree that most individual components may be known in literature, but we believe the following task-specific adaptations and system-level contributions are non-trivial:\n\n- **Unified system for audio-video understanding and generation:**\nJavisGPT is the first framework that jointly handles instruction-driven and generalized audio-visual understanding and generation within a multi-turn dialogue setup, with a shared backbone and efficient training pipeline.\n\n- **Empirical validation and design insights:**\nAll key components are supported by systematic experiments and ablations, which also provides some insightful Findings. For example, we observe that comprehension task improves generation quality, but not vice versa. We also discuss the architectural implications of this asymmetry, offering perspectives that may inform future model design in Appendix A.1.\n\n- **[Minor] Some components are not exactly the same as existing works**:\nModules such as SyncFusion and hierarchical queries are not direct copies of prior designs, but adapted to solve AV-specific challenges, i.e., capturing and maintaining the frame-level audio-video synchronization (details refer to our response to Q1). \n\nAs noted by the other reviewers, the system design, model implementation, dataset curation, and emripical findings provide **practical value and meaningful inspiration for future research** in the field.\n\n> **W2**:  The new 200 k “JavisInst‑Omni” dialogues are synthetically generated from GPT‑4 with task‑specific prompts, following LLaVA‑v1.5 and NExT‑GPT‑Lite, rather than collected from humans.\n\n**Ans**: We acknowledge the reviewer’s concern and also view this as the potiential limitation. However, while our JavisInst-Omni data is generated via GPT-4o, we argue that well-designed synthetic data can provide scalable and diverse supervision, especially for multimodal tasks where annotation is costly. This approach has proven effective in ShareGPT4Video[1], and our results show that it significantly improves instruction-following ability across modalities.\nIn the future, we plan to incorporate human annotations (such as preference data) to further enhance both the fidelity and diversity of the dataset.\n\n\n> **Q1**: How does SyncFusion differ fundamentally from earlier audio–video fusion layers such as the Q-Former/FiLM approaches used in AVFormer and NExT-GPT? Where is the genuine innovation beyond frame-aligned cross-attention?\n\n**Ans**: We hope to clarify the insights behind SyncFusion as follows:\n\n1. **Structural difference from Q-Former**: Q-Former uses learnable query tokens to extract and compress information from audio and video modalities. This process often introduces *training difficulty and information loss*[2], which can hinder effective audio-video synchronization. In comparison, SyncFusion directly injects audio information into existing visual tokens, enabling simple yet effective frame-level alignment, as supported by the quantitative results in Table 4.\n\n2. **Insights beyond frame-aligned cross-attention**: As illustrated in Figure 3, SyncFusion enables the construction of a *new modality*, where the resulting *SyncAV token* $e_{i,j}^{t} \\in \\mathbb{R}^{C}$ represents a sounding event localized at the $i$-th row, $j$-th column visual patch of the $t$-th frame. This structured semantic grounding is not present in earlier fusion approaches.\n\n3. **Avoiding redundancy**: As discussed in our response to Reviewer SwFQ's Q1, SyncFusion not only captures fine-grained AV synchronization but also avoids the redundancy issues seen in bidirectional or token-heavy fusion methods, further demonstrating its novelty and task-specific effectiveness.\n\n> **Q2**: If the temporal-spatial prior query is removed—keeping only JavisQuery and JavisCond—how much do synchrony and generation quality drop? Please provide an ablation isolating its contribution.\n\nThansks for the comment. As shown in the table below, removing the temporal-spatial prior query has minimal impact on overall generation quality and consistency, but leads to a slight drop in audio-visual synchrony. This result aligns with our motivation: the temporal-spatial prior primarily serves to enhance the temporal and spatial alignment of generated audio and video content. \n\n| Strategy          |  Quality (V/A) ↓ |    Consistency (TV/TA/AV) ↑   | Synchrony ↑ |\n|--------------------|:---------:|:-----------------:|:---------:|\n| w/o ST-Prior Query | 319.3/**7.6** | **0.146**/0.178/**0.203** | 0.150     |\n| w/ ST-Prior Query  | **317.5**/**7.6** | 0.145/**0.180**/0.202 | **0.157**     |\n\n\n> **Q3**: Could the frozen DiT be replaced with a lighter autoregressive or VQ-GAN decoder without retraining the LLM adapters? Any preliminary experiments or rationale?\n\n**Ans**: Probably not. The latent space output by the LLM adapters must align with the input space of the decoder; switching from a frozen DiT to another autoregressive or VQ-GAN decoder would break this compatibility and cannot be applied directly. One possible workaround is to retrain the adapters, or alternatively, use an agent-based approach that first generates a text caption from the instruction and then feeds it to the decoder. However, this is not recommended. Although enabling quick adaptation and application, this paradigm hurts the unified nature of the entire framework.\n\n---\n\n[1] Chen L, Wei X, Li J, et al. Sharegpt4video: Improving video understanding and generation with better captions[J]. Advances in Neural Information Processing Systems, 2024, 37: 19472-19495.\n\n[2] Yao L, Li L, Ren S, et al. Deco: Decoupling token compression from semantic abstraction in multimodal large language models[J]. arXiv preprint arXiv:2405.20985, 2024."}, {"review_id": "PsgyeNsDef", "reviewer": "Reviewer_SniZ", "summary": "This paper constructs JavisGPT, a unified MLLM for joint audio-visual understanding and generation by cascading audio-visual LLM with JavisDiT. The paper includes training recipes for different stages and provides JavisInst-Omni, an audio-visual text dialogue dataset of over 200,000 samples synthesized by GPT-4o. Experiments show that JavisGPT outperforms existing omni models on audio-visual understanding and generation benchmarks, especially in complex temporal synchronization scenarios.", "questions": "* Can the JavisGPT support multi-turn audio-visual understanding and generation through specific data and architectural designs?\n* Does the cascading architecture lead to bottlenecks in audio-visual generation? Are there any defects in efficiency?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n* Constructing a joint audio-visual understanding and generation system that achieves promising results on corresponding benchmarks is an extremely challenging yet profoundly significant task. It requires complex system design and overall optimization, which I deem highly non-trivial. Thus, the paper makes solid contributions.\n* The authors' SyncFusion design alleviates the audio-visual asynchrony issue in previous dual-branch audio-visual methods, demonstrating practical value.\n* The paper describes the architectural design, training schemes, relevant data (e.g., JavisInst-Omni), and implementation code, which positively promotes the development of the entire field.\n\n### Weaknesses\n\n* Strictly speaking, the proposed model is not the first joint audio-visual understanding and generation model—similar architectures and concepts have been presented in AV2AV [1] and MultiDialog [2], which diminishes the work's originality.\n* The authors primarily focus on single-turn \"video-audio\" pair understanding and generation. Exploring multi-turn \"video-audio\" scenarios would significantly enhance the work's significance.\n* The model employs a cascading approach with an external DiT rather than an end-to-end unified architecture for constructing the joint understanding-generation model, which diminishes part of the work's significance.\n\n[1] AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation\n[2] Let’s Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation", "was_revised": true, "final_justification": "The authors have comprehensively addressed all my questions. However, architecturally, I believe an end-to-end architecture might be more meaningful to the community. Therefore, I maintain my current positive score.", "rebuttal": "> **W1**: Strictly speaking, the proposed model is not the first joint audio-visual understanding and generation model—similar architectures and concepts have been presented in AV2AV and MultiDialog, which diminishes the work's originality.\n\n\n**Ans**: We thank the reviewer for this comment. While AV2AV, MultiDialog, and the proposed JavisGPT all involve audio-visual modalities, our originality lies in the following aspects:\n\n1. **Conceptual motivation**: AV2AV and MultiDialog are limited to the specific task of talking-head generation with speech synthesis. In comparison, JavisGPT extends the paradigm to general natural audio-visual understanding and generation, which is more comprehensive and challenging, and offers broad application potential.\n2. **Architecture**: AV2AV relies on a Transformer encoder-decoder for style transfer, while MultiDialog adopts an autoregressive language-speech model for conversation but completely discards textual input and output. Only JavisGPT employs the more mainstream encoder-LLM-decoder architecture, supporting free-form inputs across text, audio, and visual modalities, and enabling both textual dialogue and audio-visual generation. This design offers superior extensibility and generalization capability.\n\nWe will incorporate this discussion into the revised manuscript to better highlight our originality and novelty.\n\n> **W2**: The authors primarily focus on single-turn \"video-audio\" pair understanding and generation. Exploring multi-turn \"video-audio\" scenarios would significantly enhance the work's significance.\n\n> **Q1**: Can the JavisGPT support multi-turn audio-visual understanding and generation through specific data and architectural designs?\n\n**Ans**: We would like to clarify that JavisGPT is inherently capable of multi-turn audio-visual understanding and generation. Architecturally, this is enabled by formatting dialogue history into structured prompt templates (e.g., `<user>...</user><assistant>...</assistant>` repeated over turns), which allows the model to retain and use context. Examples of multi-turn interactions are shown in Fig. 1 and Fig. 9.\n\nIn our training set, we incorporated a wide range of multi-turn audio-visual scenarios into the training corpus JavisInst-Omni (Fig. 5), including both understanding and generation tasks. Furthermore, Fig. 6 reports human evaluation results conducted on interleaved multi-turn dialogues, further validating the the effectiveness of JavisGPT beyond single-turn settings.\n\n> **W3**: The model employs a cascading approach with an external DiT rather than an end-to-end unified architecture for constructing the joint understanding-generation model, which diminishes part of the work's significance.\n\n> **Q2**: Does the cascading architecture lead to bottlenecks in audio-visual generation? Are there any defects in efficiency?\n\n**Ans**: We thank the reviewer for raising this important point, which is indeed a critical consideration, and we also have discussed it in detail in Section 6 and Appendix A.1 of the paper. Below, we summarize the key arguments:\n\n1. **Effectiveness**: Under current technological constraints, the LLM+DiT framework adopted by JavisGPT is the most robust and high-performing approach[1-2]. A fully end-to-end, autoregressive multimodal architecture would require mature multimodal tokenizers, extremely large-scale training data, and highly stable training infrastructure[3-4]. Those conditions are not yet sufficiently mature in the audio-visual field. In practice, such an end-to-end model may underperform compared to our cascaded framework.\n\n2. **Efficiency**: Contrary to potential concerns, the current cascaded design is in fact efficient. As audio-visual tokens are highly dense and numerous[3-4], a fully autoregressive, token-by-token generation process would severely bottleneck inference speed, which is well-documented in works such as SelfTok[4]. In comparison, JavisGPT’s use of a query-based interface between the LLM and DiT—similar to MetaQuery[2], enabling more efficient and scalable generation.\n\n3. **Architectural perspective**: From an LLM-centric viewpoint, the DiT can be seen as a large audio-visual decoder (analogous to a text detokenizer). Even in a fully autoregressive setup, a DiT-like component would likely still be necessary to maintain generation quality[3-5]. Therefore, we believe the current JavisGPT framework remains both practical and conceptually insightful for future developments.\n\n---\n\n[1] Wu S, Fei H, Qu L, et al. Next-gpt: Any-to-any multimodal llm[C]//Forty-first International Conference on Machine Learning. 2024.\n\n[2] Pan X, Shukla S N, Singh A, et al. Transfer between modalities with metaqueries[J]. arXiv preprint arXiv:2504.06256, 2025.\n\n[3] Tong S, Fan D, Zhu J, et al. Metamorph: Multimodal understanding and generation via instruction tuning[J]. arXiv preprint arXiv:2412.14164, 2024.\n\n[4] Wang B, Yue Z, Zhang F, et al. Discrete visual tokens of autoregression, by diffusion, and for reasoning[J]. arXiv e-prints, 2025: arXiv: 2505.07538.\n\n[5] Open AI. Introducing 4o image generation. https://openai.com/index/\nintroducing-4o-image-generation/, 2025.\n\n---\n\n## Ethical Statement\n\nReviewer SniZ has flagged our submission with Major Ethical Concern on both human and data issues: \n\n> Major Concern: Improper research involving human subjects, Major Concern: Data privacy, copyright, and consent\n\nWe hope to clarify that:\n1. The human evaluation study involved three volunteers providing preference judgments. No sensitive personal data was collected. All participants gave informed consent, and institutional ethics approval was not required under our institution's policy.\n2. The proposed JavisInst-Omni dataset is constructed entirely from publicly available data sources. It does not involve any private or proprietary content, human annotation, or personal information. There are no concerns regarding data privacy, copyright, or consent."}]

Author rebuttal:

Meta-review: The paper proposes a new MLLM for synchronized AV understanding and generation based on strong spatio-temporal alignment. It proposes a  three-stage training pipeline for synchronized generation as well as a new large-scale dataset for complex audio-video-text interactions based on QA instances from existing models such as VideoLlama2. The model shows good results across various benchmarks.

The paper was considered by four reviewers with the following ratings: SA - BA - BA - BA 
 
The reviewers mainly criticised the limited conceptual novelty, but highlighted the extensive evaluation, the solid model design, as well as the option of having access to this as a public model and dataset.  Most weaknesses raised by reviewers have been addressed in the rebuttal.

The AC follows the consensus of the reviewer voting and recommends accepting the paper.
The AC would encourage the authors to integrate the findings of the rebuttal in the CR version of the paper.
Decision: Accept (spotlight)