Title: JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multi-Modality

Large Language Model

Souding-Video Comprehension and Generation
Abstract: This paper presents JavisGPT, the first unified multimodal large language model (MLLM) for joint audio-video (JAV) comprehension and generation. JavisGPT has a concise encoder-LLM-decoder architecture, which has a SyncFusion module for spatio-temporal audio-video fusion and synchrony-aware learnable queries to bridge a pretrained JAV-DiT generator. This design enables temporally coherent video-audio understanding and generation from multimodal instructions. We design an effective three-stage training pipeline consisting of multimodal pretraining, audio-video fine-tuning, and large-scale instruction-tuning, to progressively build multimodal comprehension and generation from existing vision-language models. For instruction tuning, we construct JavisInst-Omni, a high-quality instruction dataset with over 200K GPT-4o-curated audio-video-text dialogues that cover diverse and multi-level comprehension and generation scenarios. On JAV comprehension and generation benchmarks, our experiments show that JavisGPT outperforms existing MLLMs, particularly in complex and temporally synchronized settings.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper constructs JavisGPT, a unified MLLM for joint audio-visual understanding and generation by cascading audio-visual LLM with JavisDiT. The paper includes training recipes for different stages and provides JavisInst-Omni, an audio-visual text dialogue dataset of over 200,000 samples synthesized by GPT-4o. Experiments show that JavisGPT outperforms existing omni models on audio-visual understanding and generation benchmarks, especially in complex temporal synchronization scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n* Constructing a joint audio-visual understanding and generation system that achieves promising results on corresponding benchmarks is an extremely challenging yet profoundly significant task. It requires complex system design and overall optimization, which I deem highly non-trivial. Thus, the paper makes solid contributions.\n* The authors' SyncFusion design alleviates the audio-visual asynchrony issue in previous dual-branch audio-visual methods, demonstrating practical value.\n* The paper describes the architectural design, training schemes, relevant data (e.g., JavisInst-Omni), and implementation code, which positively promotes the development of the entire field.\n\n### Weaknesses\n\n* Strictly speaking, the proposed model is not the first joint audio-visual understanding and generation model—similar architectures and concepts have been presented in AV2AV [1] and MultiDialog [2], which diminishes the work's originality.\n* The authors primarily focus on single-turn \"video-audio\" pair understanding and generation. Exploring multi-turn \"video-audio\" scenarios would significantly enhance the work's significance.\n* The model employs a cascading approach with an external DiT rather than an end-to-end unified architecture for constructing the joint understanding-generation model, which diminishes part of the work's significance.\n\n[1] AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation\n[2] Let’s Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation\n\nQUESTIONS:\n* Can the JavisGPT support multi-turn audio-visual understanding and generation through specific data and architectural designs?\n* Does the cascading architecture lead to bottlenecks in audio-visual generation? Are there any defects in efficiency?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper present JavisGPT, an encoder‑LLM‑decoder system that unifies synchronized audio‑video (AV) comprehension and generation. Key ingredients include:\n\n(1) SyncFusion: a cross‑attention module that merges frame‑aligned audio features into video patches, yielding SyncAV tokens for fine‑grained spatio‑temporal reasoning.\n\n(2) Hierarchical queries (JavisQuery / JavisCond / ST‑prior) that map LLM hidden states to a frozen JAV‑DiT diffusion generator, enabling one‑shot, temporally coherent AV synthesis. \n\n(3) A three‑stage training pipeline—multimodal pre‑train, AV fine‑tune, and instruction tuning—driven by the 200 k‑dialogue JavisInst‑Omni dataset .\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n(1) Covers eight comprehension benchmarks, text‑to‑AV generation metrics, and a human study on interleaved dialogues; baseline list is current and competitive.\n\n(2) Build an model that simultaneously understands and generates fully synchronized video and audio with a shared backbone.\n\n\n**Weaknesses**\n\n(1) Limited conceptual novelty. Most of JavisGPT’s building blocks already exist in the literature and the paper does not clearly articulate what is fundamentally new beyond their specific combination:  SyncFusion is essentially a Q‑Former/FiLM‑style cross‑attention layer applied to frame‑aligned audio features; similar audio‑video fusion schemes appear in AVFormer [1]. Hierarchical queries (JavisQuery / JavisCond / ST‑prior) resemble the two‑level adapter hierarchy of Control‑Net and Uni‑Adapter, while the frozen DiT decoder mirrors Make‑A‑Video and VideoComposer. The three‑stage training recipe (multimodal pre‑train → task fine‑tune → instruction tuning) is standard practice inherited from LLaVA, MiniGPT‑4, and NExT‑GPT.  \n\n(2) The new 200 k “JavisInst‑Omni” dialogues are synthetically generated from GPT‑4 with task‑specific prompts, following LLaVA‑v1.5 and NExT‑GPT‑Lite, rather than collected from humans.\n\n[1] Seo P H, Nagrani A, Schmid C. Avformer: Injecting vision into frozen speech models for zero-shot av-asr[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 22922-22931.\n\nQUESTIONS:\n1. How does SyncFusion differ fundamentally from earlier audio–video fusion layers such as the Q-Former/FiLM approaches used in AVFormer and NExT-GPT? Where is the genuine innovation beyond frame-aligned cross-attention?\n\n2. If the temporal-spatial prior query is removed—keeping only JavisQuery and JavisCond—how much do synchrony and generation quality drop? Please provide an ablation isolating its contribution.\n\n3. Could the frozen DiT be replaced with a lighter autoregressive or VQ-GAN decoder without retraining the LLM adapters? Any preliminary experiments or rationale?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article introduces JavisGPT, which is a unified MLLM specifically designed for audio-video coherent understanding and generation. It adopts an encoder-LLM-decoder architecture based on Qwen2.5-VL, BEAT, and pre-trained JAV-DiT generator, and bridges it through the SyncFusion module. Meanwhile, the paper constructs and uses JavisInst-Omni, which is an internal dataset consisting of 200K GPT-4o audio-video-text dialogues. In the understanding and generation benchmark tests of the paper, especially in the time synchronization task, JavisGPT's performance is significantly superior to that of previous MLLMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrength \n- This study introduces a new instruction dataset, which covers a wide range of real-world scenarios.\n- SyncFusion effectively aligns fine‑grained audio and video features, enabling coherent sounding‑video generation from natural instructions.\n- The three-stage training pipeline is well-conceived, offering a systematic way to incrementally build and refine the model's complex multimodal capabilities.\n\nWeakness\n- The human evaluation in this study involved only three volunteers, which may introduce bias due to the small sample size and limited diversity of assessors. Future work could benefit from a larger and more diverse group of evaluators to enhance the reliability and generalizability of the findings.\n- The paper introduces heavy components like JAV-DiT and SyncFusion, which increase computational complexity and may hinder real-time or resource-constrained deployment. Though some comparisons with alternative methods are included, the scope is limited and does not cover simpler options like mlp. Thus, it remains unclear if the performance gains justify the higher computational costs and latency.\n\nQUESTIONS:\n- In addition to the concerns raised, the performance gains from SyncFusion appear relatively modest in table 4, and the reduced latency seems heavily tied to using fewer encoded AV tokens. This doesn’t convincingly show its superiority over Bidirectional Cross-Attention. \n- Could the authors provide additional experiments or insights to clarify where SyncFusion outperforms Bidirectional Cross-Attention, both in accuracy and efficiency?\n- Furthermore, I’m curious about how compressing audio-visual tokens impacts speed and other metrics. specifically:\nDoes cutting down on AV tokens significantly boost inference speed?\nAre there hidden costs, like a drop in accuracy or other performance trade-offs?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n# Summary\n\nThis paper proposes a novel model for unified understanding and generation of synchronized audio-video. The model is built upon the Qwen2.5-VL-7B-Instruct backbone. For encoding, images are processed using Qwen2.5-VL’s visual encoder, while audio is encoded with BEATs. An additional SyncFusion module, based on cross-attention, is introduced to obtain synchronized audio-video features, with its output dimensionality aligned to that of the encoded video features. These fused features are then fed into the LLM backbone. For generation, multiple query tokens are provided to the LLM backbone, which then produces latent conditions for an external DiT module responsible for synchronized audio-video generation. The output conditions are aligned with the T5-XXL text encoder, and the DiT module remains frozen during training. The authors further propose a hierarchical query token approach, where the output conditions of these additional queries are aligned with the Spatial-Temporal Prior encoder of JavisDiT. The final loss function comprises four terms: (1) semantic alignment loss, (2) ST-prior alignment loss, (3) denoising objective (not used during pre-training), and (4) NTP objective. The model is trained in three stages: (1) a pre-training stage to align audio inputs and multimodal outputs, (2) an audio-video fine-tuning stage to enhance synchronized audio-video comprehension and generation, and (3) instruction-tuning to improve instruction-following and reasoning abilities. Additionally, the authors introduce the JavisInst-Omni dataset.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\n1. This work represents a significant milestone in the field of sounding video understanding and generation.\n2. The technical contributions are substantial and introduce meaningful innovations.\n3. The experiments are well-designed, and the findings are both solid and interesting.\n4. JavisGPT demonstrates impressive performance.\n5. The ablation studies are thorough and provide valuable insights.\n\n# Weaknesses\n\n1. Why did the authors choose BEATs (2022) as the audio encoder, instead of adopting more recent advancements such as WavTokenizer (2024)[1]?\n2. It would be beneficial to provide more details about the evaluation metrics for audio-video generation, perhaps in the appendix.\n3. What are the benefits of cross-attention mechanism (SyncFusion) compared with interleaved concatenation ([Vclip0, Aclip0, Vclip1, Aclip1, ...]) for audio-video synchrony?\n\n[1] wavtokenizer: an efficient acoustic discrete codec tokenizer for audio language modeling\n\nQUESTIONS:\n1. **Lines 217–220**: This experimental finding is quite interesting. I am very curious about it. Generally, for video generation models, a two-stage approach involving prompt engineering (PE) and DiT is used: the PE stage leverages a language model or MLLM to generate a caption for the target video, and then DiT uses a text encoder to obtain the conditions for video generation. Although some recent methods have attempted to bypass the PE stage by directly generating such conditions with an LLM, none have shown superior video generation performance compared to the traditional PE+DiT approach. Therefore, the finding in this paper that JavisGPT outperforms JavisDiT is particularly intriguing. Can you provide more insights into this? What are the key factors that allow the latter solution to surpass the PE+DiT approach? How do you ensure that, even with only a vague textual condition (and not a clear target video caption), the performance of direct condition generation can exceed that of PE+DiT? Or what are the reasons for which the previous non-PE approaches fail?\n\n2. **Lines 109–114**: The authors freeze JavisDiT for the sake of training efficiency and simplicity. What would happen if JavisDiT were also trained (i.e., unfrozen)? Could this potentially further improve performance? What are your thoughts on the possible training stability issues when unfreezing JavisDiT, especially in terms of avoiding significant degradation in DiT performance? (I know that JavisDiT is frozen in this work, but I am very curious about this.)\n\n3. When will this model be open-sourced?\n\nThis paper is impressive. I am willing to increase my score if the weaknesses and my questions are addressed.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-15"}