Title: Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLM
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video Understanding

Video Temporal Understanding
Abstract: Humans naturally understand moments in a video by integrating visual and auditory cues. For example, localizing a scene in the video like “A scientist passionately speaks on wildlife conservation as dramatic orchestral music plays, with the audience nodding and applauding” requires simultaneous processing of visual, audio, and speech signals. However, existing models often struggle to effectively fuse and interpret audio information, limiting their capacity for comprehensive video temporal understanding. To address this, we present TriSense, a triple-modality large language model designed for holistic video temporal understanding through the integration of visual, audio, and speech modalities. Central to TriSense is a Query-Based Connector that adaptively reweights modality contributions based on the input query, enabling robust performance under modality dropout and allowing flexible combinations of available inputs. To support TriSense's multimodal capabilities, we introduce TriSense-2M, a high-quality dataset of over 2 million curated samples generated via an automated pipeline powered by fine-tuned LLMs. TriSense-2M includes long-form videos and diverse modality combinations, facilitating broad generalization. Extensive experiments across multiple benchmarks demonstrate the effectiveness of TriSense and its potential to advance multimodal video analysis.


Peer reviews:
comments: SUMMARY:
The authors design a video understanding model integrating audio, speech and visual modalities, build a three-modal caption dataset for training, and achieved promising results in temporal video understanding tasks, e.g., Moment Retrieval.

STRENGTHS AND WEAKNESSES:
### Strengths
* Achieve promising results on temporal video understanding benchmarks.
* Make substantial data contributions, which are significant for omni-model exploration.

### Weaknesses

* The authors focus only on simple and basic benchmarks/tasks, without considering implications for general video or audio-visual understanding.
* The authors claim existing models fail to fuse audio information but doesn’t explain why. The proposed architecture resembles prior designs, suggesting the key contribution lies in the dataset.
* The authors introduce audio/speech modalities without ablation studies or case demonstrations to validate their impact on temporal video understanding.

QUESTIONS:
* Why do the authors only focus on temporal video understanding tasks rather than more general video understanding benchmarks (e.g., VideoMME)?
* The design of query-based connector to accommodate multi-modal inputs has been explored in prior work (e.g., Video-LLaMA). Is it appropriate to claim this as a core technical contribution at the model architecture level?
* The authors aim to enhance video understanding (especially temporal tasks like Moment Retrieval and Segment Caption) by integrating audio-visual information, but lack ablation studies to demonstrate whether Speech/Audio modalities truly improve task performance.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents TriSense, a novel triple-modality large language model with a dynamically adaptive Query-Based Connector that enables robust video temporal understanding across arbitrary modality combinations, supported by the TriSense-2M dataset, achieving state-of-the-art performance in segment captioning and moment retrieval under diverse modality dropout scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\nThe authors propose a groundbreaking adaptive weighting framework (Query-Based Connector) that dynamically reweights modalities under arbitrary missing inputs (vision/audio/speech), validated by TriSense-2M—the only dataset explicitly supporting modality dropout (2M samples), with best performance degradation under partial modality absence.\n\nWeaknesses\n1. The modal has computational efficiency concerns, the training demands are high: Stage 3 requires 16×A100 GPUs for 7 days, and also inference uses 64–128 frames per video, costing 8–10 hours for 11.4K test samples. While Slot-Based Compression reduces token counts, the LLM backbone and temporal modeling remain resource-intensive. No lightweight variants or quantization strategies are explored, hindering edge-device deployment.\n2. Potential bias exists in the dataset. TriSense-2M relies on GPT-generated captions for its Generator or Judger training. Manual filtering (500 samples/batch) is sparse, potentially retaining inconsistent samples and the paper lacks analysis of bias impacts.\n3. The modal in this paper underperforms in visual-only settings due to its multimodal optimization focus. This suggests the model prioritizes cross-modal synergy over single-modality refinement, limiting applications where audio/speech is unavailable. The authors acknowledge this but offer no mitigation strategy.\n\nQUESTIONS:\nTypo: In table 4, the second-best results are not underlined in ActivityNet-Caption.\n\n1.\tIn table 4, is the experiment only related to the number of frames? Can you give the results of other quantitative experiments such as inference speed with a generic scenario dataset?\n2.\tIn Section 3, TriSense-2M uses GPT-o1 for Judger training, but GPT-o1's closed-source nature limits reproducibility. Do you have tried to use open-source LLMs as an alternative? Can you give the effect of using different LLMs on the data?\n3.\tIn Section 3, have you calculated the percentage of missing modal data, and what specific modal training data affects the final model performance results?\n\nLIMITATIONS:\nThe computational resources consumed by the model are too large, the experiment lacks interpretability, only the number of frames used is compared in the data comparison, and not the impact of more factors such as training data on the experiment, more adequate experiments and the same amount of data are needed for comparison to make the paper more convincing.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel triple-modality large language model for detailed video understanding that considers visual, audio and speech modalities. The proposed model incorporates a query-based connector which can automatically fuse multi-modal features. Besides, the authors collect a large-scale dataset comprising 2 million samples with synchronized video, audio and speech. The experimental results and the downstream applications demonstrate the effectiveness of the method and dataset.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n1. This paper studies a very interesting problem which seeks to build an omni-modal large language model for comprehensive multi-modal understanding. \n2. The proposed TriSense-2M dataset is useful and contains various types of captions including AVS, AV, VA captions, which can support a wide range of future studies. \n3. The proposed query-based connector in the TriSense model is novel, as it extends the previous Q-former-based connector to support three modalities: audio, visual, and speech for multi-modal tasks.\n\n**Weakness:**\n1. The overall quality of the proposed TriSense-2M dataset is not very clear. Could you provide some randomly picked examples from the dataset?\n2. According to the recommendation of the NeurIPS conferences, the authors should discuss the limitation of their proposed models and potential future improvements.\n3. As mentioned in Appendix B, in the dataset construction pipeline, only 10,000 and 3,000 samples are used to train the generator and the judger, respectively. This raises my concerns about their captioning and judging capabilities for reliable data annotation.\n\nQUESTIONS:\n1. Could you provide a detailed explanation of the feature shapes for the vision, audio, and speech modalities? Additionally, how many tokens are ultimately fed into the large language model?\n\nLIMITATIONS:\nNo. In line 273, the authors find that “model shows slightly lower performance on visual-only moment retrieval compared to state-of-the-art vision models”, future work could explore methods for balancing the multi-modality question answering performances.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces TriSense, a multimodal large language model architecture designed for comprehensive video temporal understanding, integrating visual, audio, and speech modalities. The core novelty is a Query-Based Connector that adaptively weights and fuses modalities based on the input query, enhancing robustness to missing or irrelevant modalities. The authors also present TriSense-2M, a curated, large-scale dataset of over 2 million annotated long-form video segments incorporating variable modality combinations. Extensive evaluations across segment captioning and moment retrieval tasks on both TriSense-2M and several public benchmarks show that TriSense outperforms prior models, notably excelling in settings with partial modality availability. Ablation studies further substantiate the contributions of the adaptive connector and dataset design.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n* The paper is clearly written and easy to follow, with well-organized sections and clear explanations of both the method and experiments.\n* The idea of a Query Based Connector to dynamically reweight contributions from different modalities is intuitive.\n* The introduction of the new TriSense 2M dataset is a valuable contribution to the community.\n* The proposed method achieves significant performance gains over strong baselines across multiple benchmarks, demonstrating its practical effectiveness.\n\n Weaknesses\n* Limited novelty in modality fusion techniques. While the Query Based Connector is positioned as the key modeling innovation of the paper, its design closely resembles existing approachesp[1,2]. Specifically, the query based cross attention mechanism is highly similar to the instruction-aware QFormer proposed in InstructBLIP and X-InstructBLIP[1]. In addition, the dynamic reweighting of modalities resembles the self-gated query fusion strategy used in CREMA[2]. As such, the proposed connector appears to be more of a combination of these two prior techniques rather than a fundamentally novel modality fusion method. This overlap limits the originality of the technical contribution.\n* Missing citations and related work discussion. Given that the Query Based Connector is the central technical innovation of the paper and even emphasized in the abstract, the lack of a thorough discussion of related work on modality fusion is a notable weakness. A more comprehensive related work section discussing these and clarifying how the proposed connector differs or improves would strengthen the paper and help position it more clearly in the landscape of multi modal fusion research.\n\nReferences\n\\[1] X InstructBLIP: A Framework for Aligning Image 3D Audio Video to LLMs and its Emergent Cross modal Reasoning 2023\n\\[2] CREMA: Generalizable and Efficient Video Language Reasoning via Multimodal Modular Fusion ICLR 2025\n\nQUESTIONS:\nMy main concern is the novelty of the modeling, as I do not see substantial innovation in the design. But I appreciate the introduction of a high-quality AVS dataset, which is a valuable contribution to the community. Overall, I prefer to be borderline; but, since a strictly neutral rating is not available, I lean slightly positive and therefore give a borderline accept. I will make my final decision after reading the authors’ rebuttal.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}