Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLM
Authors: Zinuo Li
Xian Zhang
Yongxin Guo
Mohammed Bennamoun
Farid Boussaid
Girish Dwivedi
Luqi Gong
Qiuhong Ke
Venue: NeurIPS 2025 poster ()
Keywords: Video Understanding
Video Temporal Understanding
Primary area: applications
TL;DR: We introduce a new triple-modality MLLM TriSense, which achieves comprehensive understanding of video moments by adaptively integrating visual, audio, and speech information. To support this, we propose a newly constructed dataset TriSense-2M.

Abstract: Humans naturally understand moments in a video by integrating visual and auditory cues. For example, localizing a scene in the video like “A scientist passionately speaks on wildlife conservation as dramatic orchestral music plays, with the audience nodding and applauding” requires simultaneous processing of visual, audio, and speech signals. However, existing models often struggle to effectively fuse and interpret audio information, limiting their capacity for comprehensive video temporal understanding. To address this, we present TriSense, a triple-modality large language model designed for holistic video temporal understanding through the integration of visual, audio, and speech modalities. Central to TriSense is a Query-Based Connector that adaptively reweights modality contributions based on the input query, enabling robust performance under modality dropout and allowing flexible combinations of available inputs. To support TriSense's multimodal capabilities, we introduce TriSense-2M, a high-quality dataset of over 2 million curated samples generated via an automated pipeline powered by fine-tuned LLMs. TriSense-2M includes long-form videos and diverse modality combinations, facilitating broad generalization. Extensive experiments across multiple benchmarks demonstrate the effectiveness of TriSense and its potential to advance multimodal video analysis.

Watch and Listen: Understanding Audio- Visual-Speech Moments with Multimodal LLM
Zinuo Li', Xian Zhang', Yongxin Guo?, Mohammed Bennamoun Farid Boussaid', Girish Dwivedi', Luqi Gong"; Qiuhong Ke
https //gi thub. com/zinuoi L/TriSense University of Western Australia Alibaba Group Zhejiang Laboratory *Monash University
AVS Segment Captoning
Segment Cactooinng
AV Segment Capt Captining oning
Visucl Only Segment Coptionin what vieud evert Kmapowning the video.
4 condswing from hot audio
thm viuui speach eveet heppwingg
7P
inserily the yiuul sound nvint and
7%
Viss spwech cupects
locuaiiy on what's viible and spokent
werking while mask
man weosing loce most toli
coeoinng obo spenear bokkuu snocks
bodig oonddggodddd sound whah
Eokiou mentoning
onother pessonatoo obout the sponsor
been
mos weering face toling
foce mask tolking
leon butn lyving
lviing snocko
AVS Moment Retrival
VS Momeent Ree Fervel
AV Mormeni Reirievol
Visual Visual-Onll Moment Roft ciall
vectrin while mosk upedling
vosling local oreas in Tolays, lilming the
Fonsor otber with sheed noke and converstiorn bochground. Anothar iily bike the while on speats ideecill""
belennine the preciss fine seconda content by wolldng the video, provide Information.
utoou Bokktu
traclggrond sourdd which Ihay
shope: and the recond of which
floke sour trying ln yorr.
withis video.
Speech: 35 yuan. it's lio 30 centr.
Speech: Poople to gothor in The area and this place is aso knnwn to cheepp sieel food
Speech: By the way guss this video is sporoved by boikbeu, you ga-n brow frying bowos shocks
Speech: reaily roolly recommend if you /ve never been to one, lo vial one thess load crwex, and werk one
Speech: OK guys, there's one more snepping day in the cress and Inaleed bookhrokkng
con tee
year...
1750
Noke
Soundi Stroet
Sound: Steee Noko: People Tolling
Sound:
No Foop Tolking
Figure I: TriSense supports segment captioning and moment retrieval for vidcos from audio, visual, and speech modalities, as well as any combination of them, covering total of eight different tasks.
Abstract
Humans naturally understand moments in a video by integrating visual and auditory cues. For example, localizing scene in the video like "A scientist passionately speaks on wildlife conservation as dramatic orchestral music plays, with the audience nodding and upplauding" requires simultaneous processing of visual, audio, and speech signals. However, existing models often struggic to effectively fuse and interpret audio information, limiting their capacity for comprehensive video temporal understanding. To address this, we present TriSense, triple- modality large language model designed for holistic video temporal understanding through the integration of visual, audio, and speech modalities. Central to TriSense is a Query-Based Connector that adaptively reweights modality contributions based on the input query, enabling robust performance under modality dropout and allowing flexible combinations of available inputs. To support TriSense's s multimodal capabilities, we introduce TriSense-2M, high-quality dataset of over million curated samples generated via an automated pipeline powered by fine-tuned LLMs. TriSense-2M includes long-form videos and diverse modality combinations, facilitating broad generalization. Extensive experiments across multiple benchmarks demonstrate the effectiveness of TriSense and its potential to advance multimodal video analysis.
"correspondigg authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Human understanding of real-world events is inherently multimodal: we rely not only on vision but also on spoken language and audio cues to make sense of what is happening in video. This integration allows us interpret intention, emotion, and the significance of events-gov what is seen to include what is heard and said. However, existing Multimodal Large Language Models (MLLMs) often fall short of achieving this level of nuanced video temporal understanding. While advances in vision-language modeling and temporal localization have improved language-visual alignment [11. 23, 15. 12, 38, 19, 28), most models still rely solely on visual inputs. As a result, they perform poorly on tasks requiring the integration of audio and speech particularly when one or more modalities are missing, noisy, or contextually irrelevant. This stands in contrast to human perceptual robustness and significantly limits model generalization in real-world scenarios, such as accurately localizing events or generating rich multimodal descriptions.
beyond
Challenges and Current Limitations. As highlighted above, the current state of multimodal temporal understanding remains limited. Building on these observations, two core challenges continue hinder progress in multimodal temporal understanding: 1) Insufficient and Incomplete Training Data: Cur- rent datasets are often composed of short clips and lack large-scale, fully annotated examples across audio, and speech-whice are essential for effective multimodal pretrain- ing [14, 8, 13, 12]. This scarcity hampers the development of robust. MI..Ms. Moreover, real- world videos often contain incomplete or inconsistent modality coverage, due to factors like variable record- ing setups, intentional omissions (c.g., silent footage or background music), or the natural absence of certain signals in specific scenes. When models are trained predominantly on videos with all modal- ities present, they often fail at inference time when confronted with missing or degraded inputs a common scenario in the wild. 2) Lack of Modality Adaptation: Current MLLMs are generally not equipped to assess the relative importance of each modality based on task or query context. Recent models such as LongVALE [10] and Qwen2.5-Ommi [34] attempt integrate multiple modalities but fall short in adaptivity. For instance, Long VALE compresses all modality tokens into a single represen- tation, resulting in information loss and poor handling of missing modalities. It also lacks adaptive dropout strategy, leading to unstable performance when modality availability varies. (Dwen2.5-Oma introduces temporal positional embeddings, but still fails to capture fine-grained temporal dependen- cies, limiting its effectiveness on complex moment-level tasks, as demonstrated in our experiments.
all three modalities--vsiie
Key Contributions. We argue that understanding complex moments in video requires not only broader modality coverage but also an adaptive mechanism to selectively emphasize the most relevant modalities depending on the task and query. Our approach addresses these challenges through the following key contributions:
1) We introduce TriSense-2M, large-scale multimodal dataset containing 2 million annotations. Each video instance in the dataset includes event based annotations across vision, audio, and speech modalities, with flexible combinations and natural absences of modalities. The dataset supports wide variety of scenes and includes long-form videos averaging 905 seconds-significann than those in existing tantly, queries are expressed in high-quality natural language, aligned with temporal annotations, and span diverse modality configurations to facilitate robust multimodal learning.
longer enabling deeper and more realistic temporal understanding. Impor-
2) We propose TriSense, a moment retrieval under diverse modality configurations. As depicted in Figure 1, TriSense is designed to handle multimodal video data with varying availability of vision, audio, and speech over temporal dimension. Crucially, it includes Query-Based Connector that dynamically adjusts modality weights based on the query's 's content and context. This allows the model to emphasize the most informative modalities (e.g.. prioritizing vision if most relevant) while dewn-weighting irrelevant or missing ones ones-enabling strong performance even under incomplete modality conditions.
MLLM designed for both video segment captioning and
3) We conduct extensive experiments on two core -vide segment captioning and moment retrieval aarieval-across eight modality configurations, including zero-shot evaluation on public benchmarks. TriSense achieves strong performance on both the new TriSense-2M dataset and existing benchmarks, laying a solid foundation for future research in multimodal temporal video understanding.
2

---

2 Related Work
2.1 Video Temporal Understanding MLLMS
Video temporal understanding focuses on modeling how events unfold over time within a video, enabling tasks such as moment retrieval, segment captioning, and dense video captioning. Vision- language models (VLMs) have demonstrated strong capabilities in solving real-world problems, including in zero-shot scenarios without task specific fine-tuning. However, many of these models still face challenges when it comes to understanding temporal dynamics [36, 5]. To address this, several models have been fine-tuned on video grounding datasets that emphasize temporal structure such as TimeChat [27], VTimeLLM [15]-1o enhance their temporal reasoning abilities. More recently, Mo- mentor [23] introduced a time encoder to correct errors caused by time token quantization, while VTG- LLM [12] employed specialized time tokens and temporal position embeddings to help video LLMS better capture the timing of events. In a different approach, TRACE [11] applied principles from causal language modeling propose causal event modeling for videos. It also introduced a lightweight time tower for encoding temporal information, achieving solid performance in temporal understanding.
Extending beyond lision-language modeling, Multimodal Large Language Models (MLLMs) integrate visual, audio, and speech modalities to enable richer video analysis. To improve performance, recent efforts have focused on incorporating these additional modalities. For example, LongVALE [10] compresses all modality tokens into a single token, but this leads to loss of fine-grained details such as audio context or speech intonation. Qwen2.5-Omn [34], on the other hand, introduces the Thinker-Talker architecture-a fully end-to-end multimodal system that handles text, images, audio, and video inputs. To better align audio and video along the temporal dimension, the authors propose TMRoPE, a timing-aware positional encoding technique. However, experiments show that this model still struggles with grounding in long-form videos.
Despite these advancements, many existing models are either limited to visual modalities or lack support for flexible combinations of different modalities. This restricts their effectiveness in temporal understanding tasks, particularly in scenarios where some modalities may be absent or noisy. These challenges motivate our pursuit of MLLMs that can robustly handle various modality combinations while maintaining strong temporal reasoning and general understanding performance.
2.2 Benchmarks for Temporal Understanding
The development of benchmark datasets has played a crucial role in advancing video temporal understanding. Early contributions such as DiDeMo [14] introduced natural language queries for moment localization in videos. Subsequent datasets like Charades-STr [8] and ActivityNet Captions [18] expanded on this by covering wider variety of actions and longer video durations, significantly pushing the field forward. More recently, Intern Vid2 [32] has emerged as a large-scale foundational dataset, offering 61 million andio-visualspeed annotations. However, many of these annotations are disjointed, lacking coherence across modalities, and the dataset contains substantial number of low-quality captions due to its scale.
To address these limitations, VAST-27M [3] and VALOR [2] were introduced. Both datasets offer high-quality, omni-modal (audio-visual-speech) annotations with better interrelated features than Intern Vid2, supporting comprehensive multimodal understanding for MLLMs. Nonetheless, they rely on simple concatenation of captions across modalities and do not incorporate cross-modal reasoning. Moreover, these benchmarks provide only coarse -aarssedr captions for short clips, making them inadequate for fine-grate ine-grained understanding of long-form video. Although both datasets improve on Intern Vid2, they repeat similar limitations: their annotations are not contextually integrated across modalities, and their temporal granularity too coarse for modeling nuanced event transitions in extended videos. In response to these shortcomings, Long VALE [10] was proposed, featuring 108K high-quality annotations with omni-modal coverage. While offers notable improvements, VAST-27M, VALOR, and VALE all overlook a critical issue: the dynamic presence of modalities. In real-world videos, audio, visual, and speech inputs are not always available simultaneously, raising important questions about model robustness in the face of missing modalities.
In conclusion, existing benchmarks often focus exclusively on visual information or fail to adequately support flexible multimodal integration. These limitations highlight the need for improved datasets and serve as a key motivation for our work.
3

---

Data Construction
As discussed in Sections and while some existing datasets include all three modalities-visual audio, and speech ppeech-they generally assume that these modalities are always present simultane- ously [3, 2, 10], overlooking the importance of supporting arbitrary combinations. This assumption limits the development of models that can handle missing partial inputs effectively. To address this, we introduce a new large-scale, high-quality dataset designed to support both fully multimodal and partially multimodal scenarios.
AVS Capfion Two meem che speaking and lalughing while someone else angrnng and talkngg and the ugies! thing is about to hapcen.
Visual Caption Two singing and compping hardi, speaking and loughing. Caplion
3/5
LLM Generator
Caption
LLM Judger
v Keep
Two men are singing. clapping, and taildng.
Someeone sigiing and taling, somecne yelling
One ihemm yells while
Average Score
< 3/5
Speech Caption
Filter
Vi Coption
you're
Two men are clopping and speaking syn,,, laughin One of
"This ugiiest
We employ automated framework to build dataset by leveraging madallty-speccife captions from vision, audio, and speech streams. Two large language models (LLMs) are trained for this process: Generator, which fuses the three input captions into multi-modal outputs (AVS, AV. VS), and a Judger, which evaluates the semantic quality of the generated captions. The Judger assigns an average quality score between and 5 based on alignment with the original inputs. Samples scoring 3 are retained, while those scoring < are discarded.
Figure 2:
Our dataset includes longer video durations. making it suitable for realistic and fine fine-grained temporal grounding tasks. It enables models to "watch and listen" in a human- manner, flexibly attending to available visual, auditory, and speech tion data to promote deeper video understanding and narrative generation. To support scalable and con- sistent annotation, developed fully automated data construction pipeline, shown in Figure 2.
identify relevant moments. In addition, include cap-
We begin by selecting subsets from InternVid [32] and VAST [3]
raw sources for both video content and initial captions. Each video clip is annotated with three distinct captions: visual caption that describes observable scenes and actions, an audio caption that details acoustic clements, and speech caption that transcribes spoken content. These modality-specine captions generated using specialized annotation pipelines adapted from previous works 3).
B 2-5 min [0.1%)
E min (10.28)
10-20 (83.5%)
20-30 min [4.2%]
I 30 min [1.9%]
[0.1%)
To enable reasoning across modalities, goal is to synthesize omni-modal captions that flexibly combine distinct unimodal annotations. These are crucial for training models capable of comprehensive temporal two
Video duration distribution. Most videos are 10-20 minutes long (83. (83.5%), supporting long-form tempo- ral understanding.
Figure
understanding. To generate these captions, custom- custom-trained large language models based on Qwen2.5- 72B [35]: Generator and a Judger. The Generator merges moodality-speciff captions into unified representations for three combinations: AVS (Audio- Audio-Visual-Syaa Visual-Speech). AV (Audio- Visual), and VS -peech. These captions are designed capture cross-modal interactions, such as clapping that aligns with vocal rhythm or speech that corresponds with visual context. The Judger evaluates the quality of each synthesized caption by measuring its semantic alignment with the original unimodal annotations. It assigns a quality score ranging from to 5 and filters out samples with inconsistencies, such as speech that does not relate visual actions mismatched audio-visual descriptions. To train these models, we first build high-quality reference corpus using GPT-ol [16], which is then
(Audio-Visual),

---

manually refined and filtered. From this curated set, we select 10,000 samples to train the Generator and 3,000 samples train the Judger. Further training details are provided the Appendix.
Table I: Comparison of temporal understanding datasets. Tr uniquely supports all three modalities with long video lengths and explicit handling of modality dropout.
Datasets
Annotations
Avg. len
Visual
Audio
Speech
Modality Dropout
VALOR [2]
1.32M
10s
X
VAST [3]
27M
30s
X
UnAV-100 [9]
30K
4s.1s
/
X
X
Charades-STA [8]
16K
30s
/
X
X
X
ActivityNet-Cptioon. [13]
20K
180s
X
LongVALE [10]
108K
235s
X
TriSense-2M
2M
905s
Table 2: Detailed Information of TriSense-2M. where Avg/Min/Max Duration represent the average, minimum, and maximum duration, respectively, 0-5s/ 5-10s / 10-15s, etc., represent the proportions of different duration intervals.
Total Events
Avg Duration
Min Duration
Max Duration
0-5s
5-10s
10-15s
15-20s
20-30s
1940522
6.87s
2.00s
30.00s
35.5%
46.2%
12.6%
4.2%
1.5%
The data construction pipeline processes an initial pool of million multimodal video samples containing visual, audio, and speech captions. Through multiple rounds of generation, evaluation, and filtering, the Judger retains only high-quality outputs, resulting in a final dataset of 2 million samples drawn from approximately 38,000 long videos. The distribution of video durations is shown in Figure 3. The average video length is 905 seconds, nearly four times longer than that of the closest existing dataset, which averages 235 seconds [10]. This curated dataset enables robust temporal reasoning across diverse modality combinations and forms the foundation for training and evaluating our TriSense model. Overall comparisons with existing datasets provided in Table 1, and more detailed examples are included in the Appendix.
4 TriSense Architecture
The overall architecture of the TriSense model is illustrated in Figure 4. The model designed to process visual, audio, and speech information extracted from video in order to answer text-based query. Each modality is first processed by one of three specialized expert encoders [24, 26, 4]. The resulting feature representations are then passed through modality-specific projectors and integrated with the query using interactions between each modality and the query. A Query-Based Connector subsequently fuses the modality representations and adaptively adjusts their contributions based on the content of the query. This fused multimodal representation is enriched with temporal information and then input into a Large Language Model (LLM) backbone, which produces the final output. To effectively model temporal dependencies, the architecture includes dedicated Time Encoder [11] and employs tasc-specific heads to ensure outputs are temporally aligned and task-relevant.
mechanisms, allowing the model to capture fine-grained
4.1 Multimodal Information Extraction
During training, given a video V = {Fi, Fr. where T denotes the total number of frames, first uniformly select n 64 frames for further processing and controlling memory usage. For each selected frame, we record its timestamp and extract an audio segment of +1 second around the frame, resulting in sequence of audio segments A {A1 Ag, A...Ar,,,,, J. each lasting seconds. For instance, if a frame is sampled at 123.4 seconds, the corresponding audio segment spans from 122.4 124.4 seconds. We then use pretrained expert encoders to extract modality-spectie tokens: f" f f for vision, audio, and speech, repectively.
To enhance temporal awareness of the model, we encode the selected timestamp using Time Encoder [11]. Each timestamp is first tokenized into a fixed-length character sequence comprising four integer digits, decimal point, and one fractional digit, resulting
tokens per frame. For example, the timestamp [123.4] is tokenized to (0)(1) (3) (.) (4). These tokens are then embedded form time features ft). Given the high token count from the three modalities, we apply Slot-Based
5

---

lime Head
LM Heod
Large Language Model
Compress Two Twollayer MLP
Query-Based Connector
Stage2 Trainable
Mutimodia tokend (Wy+
Wo+
Staged Trminobii Mocilles
WS
Fronen Componeents
Cross Attention
Cross Attention
Vision Tokenns
Audio Tokien
Speech Tokens
f
Projector
Audio Projector
Speech Projecto
Time
Ercoder
Vision Encocer
Audio Encoder
Speech Encode
lext lakere
Text Tokenizer
Query: Locate the moment where about
60%
104h 2545
2675
125%
yerr". return timestampss
Figure 4: Architecture of the TriSense model. The model processes vision, audio, and speech via dedicated encoders and fuses them using Query-Based Connector that assigns weights based on the query. The fused output, combined with temporal embeddings, is passed an LLM for generating timestamped or textual responses.
Compression [12] as Modality Projector to reduce the dimensionality of the input. This technique compresses the vision tokens fú" audio tokens fü¹ and speech tokens fr, into fixed-length vectors of 16 tokens each, denoted as fi" S and f(=
4.2 Query-Based Connector
To more effectively integrate multimodal inputs with the query and enhance sensitivity to salient features, we introduce a Query-Based Connector that adaptively balances the contributions of each modality based on the query's content, as illustrated in Figure 4. The compressed modality features f/ 1/" and ! obtained from Section 4.1, are passed through Cross-Atter Attentoonn layers, where they interact with the encoded query representation flæ), The objective is for each attention layer to emphasize features that are most relevant the query. The outputs of these layers are denoted as query- relevant features f. S and f:a which reflect the alignment between each modality and the query,
To dynamically determine the importance of each modality in relation the query, we introduce an adaptive weighting mechanism. First, we apply global average pooling over the sequence dimension of each modality to derive compact global representations C., Ca. and c.. These vectors are concatenated and fed into single-layer MIP F()) to generate unnormalized weights We. Wa. and w,, The weights are then normalized using a softmax function to yield a valid probability distribution over the modalities, satisfying the constraint W1 + + 1. The computation is formalized below:
C.
1 Im T. Et= Xm.t.
E (v,a,8}
Um F(coo || co co E R3
m (v,
(2)
Wm exp((mm)
m (v,a,s}
(3)
expl Wm))
mn ((.a.s))
Here, Cm is the compressed modality vector after global average pooling, Tm indicates the sequence length of distinct modality m, denotes vector concatenation, and w. and wm represent unnormalized and normalized weights, respectively.
After computing the weights, we multiply the previously obtained query-relevant features by their corresponding weights and concatenate them together fuse multimodal representation. To reduce the token count, we apply slot compression (-) again to compress the tokens from triple the amount back to single scale. Finally, two-layer MLP F(-) is used for further feature

---

refinement, aligning the representation with the LLM's input dimensionality and enhancing its expressive capacity:
Xm FCFCCompccoonna(ee f wast" We :.
This adaptive fusion allows the model to emphasize the most informative modalities while reducing the influence of less relevant ones, based the specific query. The resulting representation Xmm is combined with the corresponding time embeddings f(t) (as introduced in Section 4. and passed into the LLM backbone, which uses its contextual reasoning capabilities to generate the final output.
4.3
Causal Event Prediction
To enhance the model's temporal reasoning capabilities and better align predictions with the underly- ing structure of video narratives, we employ causal event prediction, method shown to be effective in prior work [11]. This approach enables the model to reason about cause-and-effect relationships across time, predicting upcoming events based on prior context. Specifically, given a video V. we segment it into sequence of events {E1, eg, ek J. where each event ex (tx, C&) consists of timestamp t and an associated caption с₁ describing the video segment:
V = {ei, ,2, { {(l2, ex)]1 k S K}.
(5)
Our goal to predict the next event e₁ conditioned on the sequence of prior events 61 the user-provided query Q, and the multimodal features Xm produced by the Query-Baser -uared Connector:
P(ealexs-1,, Xm) P(te, c2/e1:x-1, Xm,)
(6)
To support both temporal and textual outputs, introduce adaptive head switching via special (sync) token during LLM generation. This token is appended to the vocabulary and serves as a control signal that guides the model to switch between time head and language model (LM) head, as illustrated in Figure 4. When the (sync) token is encountered, the LLM transitions between decoding modalities to generate either timestamp-aligned predictions free-form textual outputs, depending the task.
5 Experiments
In this section, we present the core experiments conducted to evaluate the performance of our proposed model. Due space constraints, implementation details, training procedures, and additional experimental results are provided in the Appendix.
5.1 Evaluation Datasets, Metrics and Baseline Models
To rigorously assess the effectiveness of TriSense, conduct evaluations two key temporal understanding tasks:
Segment Captioning (SC). This task involves generating descriptive captions that accurately summarize the events occuring throughout a video. We evaluate our model on the newly introduced TriSense-20 2M dataset, and provide additional datasets in the Appendix. The performances are reported in BLEU-4 [22], CIDEr [30], ROUGE [20] and METEOR [1] to gauge the quality and accuracy of the generated captions.
Moment Retrieval (MR). In this task, the model required to retrieve specific segments within a video that correspond to given textual query. We evaluate performance on TriSense-2M, as well as two widely used public benchmarks: Charades-STA [8] and ActivityNet-Capiine trieval effectiveness is reported using Recall @ 10,5 Recall @ IoU=0.7, and mean loU (mloU), providing a comprchensive view of the model's localization accuracy at varying overlap thresholds.
[13]. Re-
The modality combinations are set to Audio Visual-Speech (AVS), Visual-Speed (VS), Audio- Visual (AV) and Visual-Only (V). To establish solid comparative baseline, we select representative models specifically designed for Video Temporal Grounding (VTG) tasks. These include VLimeLLM [15], TimeChat [27], VTG-LLM [12], TRACE [11], along with two recent omni-modal models Long- VALE [10] and Qwen2.5-Omm [34]. While Momentor [23]. Hawkeye [31] and NumPro-FT [33] are not included in the TriSense-2M benchmark due to the unavailability of model checkpoints and their
lack of support for captioning, they are included in public benchmark evaluations according to the
7

---

Table 3: Segment Captioning results. Performance is reported on four modality settings using
BLEU-4 (B), METEOR (M), ROUGE-L (R), and CIDEr (C). Best and second-best results are in bold and underlined, respectively.
AVS-SC
VS-SC
AV-SC
V-SC
Model
B
M
R
C
M
R
B
M
R
C
B
C
VT (7B)
0.8
8.2
16.1
2,4
1.2
8.8
16.9
3.1
1.3
10.3
17.9
2.6
1.4
10.4
18.2
4,0
TimeChat
0.6
8.7
5.5
10.5
1.5
0.8
5.7
VTG-LLM (7B)
0.3
4.8
9.6
0.6
0.3
4.9
10.0
0.9
0.4
5.2
10.2
0.9
0.3
5.0
9.8
1.4
(7B)
9.0
TRACE-uni (7B)
1.1
8.2
14.7
1.4
1.5
8.3
15.1
2.2
1.6
9.5
16.3
2.3
1.3
9.9
17.6
8.8
Long VALE (7B)
1.2
4.9
2.3
10.0
20.1
2.5
11.4
21.3
5.9
1.5
11.5
0.9
Qwen2.5 1ann2.5-O0ma 5-Omii (7B)
0.8
8.8
13.1
1.7
0.8
8.6
13.1
0.8
1.2
9.8
15.1
1.3
111
10.1
14,6
1.1
TriSense (7B)
3.4
10.1
20.1
3.0
10.0
22.2
11.8
5.3
12.2
26.3
15.4
73
12.6
30.7
36.3
Table 4: Moment Retrieval results with 64 frames. Performance reported as Recall at IoU 0.5 and 0.7 across four modality settings. Best and second-best results are in bold and underlined. respectively.
AVS-MR
VS-MR
AV-MR
V-MR
Model
1oU=0.5
loU=0.7
loU=0.5
loU=0.7
loU=0.5
loU=0.7
IoU=0.5
IoU=0.7
LLM (7B)
0.09
0.28
0.14
0.23
0.08
0.41
TimeChat (7B)
0.12
0.27
0.09
0.22
0.08
0.34
0.12
VTG-LLM (7B)
0.19
0.08
0.15
0.05
0.21
0.07
0.23
0.06
TRACE (7B)
0.39
0.12
0.31
0.15
0.24
0.13
0.42
0.21
TRACE-uni (7B)
0.30
0.17
0.35
0.17
0.24
0.18
0.48
0.22
Long (7B)
0.01
0.07
0.01
0.07
0.01
0.05
Qwen2.5-Omil (7B)
0.61
0.21
0.16
0.28
0.07
0.18
0.06
TriSense (7B)
1.12
0.42
0.80
0.28
0.57
0.21
0.43
0.22
reports their official papers. By comparing our model against these baselines across diverse tasks and evaluation metrics, we aim to provide a comprehensive assessment of TriSense's ' capabilities and its advancements video temporal understanding.
5.2 Results and Analysis
Superior Performance on Omni-Modal datasets. We evaluate our performance on the proposed TriSense-2N9 2M dataset. As illustrated in Table and Table 4, TriSense consistently outperforms existing video LLMs across nearly all evaluated tasks. It also significantly surpasses latest omni-modal models like Long VALE [10] and Qwen2.5-Omm [34], particularly in the audio-viuual-spedd (AVS) setting where all three modalities are leveraged.
We observe that the model shows slightly lower performance on visual-only moment retrieval compared to state-of-the-art vision models, which is likely due to its optimization for multimodal settings rather than visual-only scenario. Also, is important to note that our model uses only 64 input frames during testing, compared to larger input sizes used in other models-such as 128 frames TRACE [11] and 100 frames in VTimeLLM [15]. Since TriSense-2M consists mostly of long videos. using fewer frames makes it more difficult for the model to achieve high accuracy in long-video moment retrieval tasks. When the videos become shorter more frames are used, gain better performances, this is also supported by the results in Table 6 and Table 7.
We also conduct experiments the public Omni-Modal benchmark Long VALE [10]. Long VALE is designed for event understanding across vision, audio, and language modalities, comprising 105,000 omni-modal events with precise temporal annotations and relation-aware captions, collected from 8,400 high-qualing long-form videos. The Omni-VTG and Omni-SC tasks named from the official report of Long VALE, which the in Table 5, our zero-shot performance on the Moment Retrieval task is comparable to LongVALE VALE's performance, even though their model trained on the same dataset. Although there a larger gap in the Segment Captioning task, we believe this is due to pattern differences in captioning styles between our Segment Captioning training data and LongVALE? VALE's Segment Captioning data. Such differences in caption patterns can lead to noticeable drops across all four evaluation metrics.
as AVS-MR and AVS-SC in our paper. As summarized

---

Table 5: Performance public Omni-Modal benchmark LongVALE [10]. "*" indicates this model is trained on the Long VALE dataset. The best and second results are highlighted in bold and underlined, respectively.
Omni-VTC (AVS-MR)
Omni-SC (AVS-SC)
Model
@@0.3
R@0.5
R@0.7
mloU
B
M
R
C
VideoChat (7B)
2.2
0.9
0.4
3.0
0.5
9.6
0.0
8.2
VideoChatGPT (7B)
4.9
2.9
0.9
5.0
0.4
14.0
0.9
5,9
VideoLLLaMA (7B)
2.5
1.1
0.3
1.9
0.9
11.5
0.1
8.9
PandaGPT (7B)
2.5
1.0
0.3
2.2
0.6
14.9
0.3
8.9
NEXT-GPT (7B)
4.3
1.9
0.7
4.0
0.4
10.2
0.0
8.1
TimeChat (7B)
5.8
2.6
1.1
5.2
1.2
16.1
1.6
10.0
VTimeLLM (7B)
7.5
3.4
1.3
6.4
1.0
14.5
1.6
5.5
VALE* (7B)
15.7
8.6
3,9
11.0
5.6
22.4
20.3
10.9
TriSense (7B)
14.8
9.3
4.7
11.2
4.8
21.9
18.8
10.4
Table 6: Zero-shot Moment Retrieval results on public benchmarks with 64 frames. "*" indicates this model uses more frames than TriSense. The top and second-best results are highlighted in bold and underlined, respectively.
Charades-STA
ActivityNet-Cappe
Model
loU=0.5
loU=0.7
mloU
loU=0.5
IoU=0.7
mloU
VLimeLLM (7B)
27.5
11.4
31.2
27.8
14.3
30.4
LLM (13B)
34.3
14.7
34.6
29.5
14.2
31.4
TimeChat* (7B)
32.2
13.4
4.6
2.0
6.9
Momentor (7B)
26.6
28.5
23.0
12.4
29.3
HawkEye (7B)
31.4
14.5
33.7
29.3
10.7
32.7
VTG-LLM (7B)
33.8
15.7
8.3
3.7
12.0
TRACE* (7B)
40.3
19.4
38.7
37.7
24.0
39.0
TRACE-uni* (7B)
43.7
21.0
41.5
38.2
24.7
39.4
NumPro-FT*s (7B)
42.0
20.6
41.4
37.5
38.8
TriSense (7B)
42.3
27.6
39.8
39.6
27.2
40.1
Zero-shot performance public Moment Retrieval benchmarks. We also evaluate TriSense in zero-shot setting on established two classical visual-only datasets, including Charades-STA. [8] and CetiittyNe-Caaptions [13]. shown in Table 6. The results show that although TriSense shows slight inferior performance in Table 4, it still achieves competitive performance in visual-only settings, showing especially higher accuracy (IoU=0.7) than others, even with less frames used.
Table 7: Ablation studies on Moment Retrieval. The top and second- best results are highlighted in bold and underlined, respectively.
AVS-MR
VS-MR
AV-MR
V-MR
Model
Frame Number
IoU=0.5
IoU=0.7
IoU=0.5
IoU=0.7
IoU=0.5
IoU=0.7
IoU=0.5
IoU=0.7
Training Stages
Stagel Only
64
0.07
0.01
0.06
0.01
0.06
0.00
0.02
0.00
Stugel42
64
0.52
0.19
0.43
0.18
0.32
0.12
0.27
0.14
Connector
Addition
0.71
0.69
0.21
0.11
0,22
Fixed Weights
0.89
0.38
0.77
0.24
0.52
0.19
0.44
0.23
Frame Number
TriSense (7B)
32
0.74
0.27
0.68
0.18
0.39
0.13
0.24
0.11
TriSense (7B)
1.12
0.42
0.80
0.28
0.21
0.43
TriSense (7B)
128
1.12
0.43
0.87
0.31
0.64
0.32
0.49
0.26
Ablation Studies. We conduct ablation experiments to assess the contribution of different compo- nents, including the training strategy, the Query-Basel Connector, and the number frames processed by the model. As shown in Table 7 and Table 8, we compare our adaptive weighting strategy against simpler alternatives. The Addition baseline directly sums modality features without weighting. The Fixed Weights baseline assigns equal weights (c.g., 0.33 each in AVS), or

---

fixed values depending on the modality pair (e.g., g., 0.5 for each active modality and 0 for inactive ones in VS/AV), and uses only the visual stream in visual-only tasks (weight of I, others set to 0). These comparisons confirm the effectiveness of our query-adaptive weighting mechanism.
Table 8: Ablation results on Segment Captioning. We analyze the effects of training stages, connector design, and input frame count across four modality settings (AVS, VS, AV, V). Metrics include BLEU-4 (B), METEOR (M), ROUGE-L (R), and CIDEr (C).
Model
Frume Number
AVS-SC
VS-SC
AV-CC
V-SC
Training Stages
Stagel
64
0.0
1.5
4.3
0.1
0.0
1.4
4.1
0.1
0.0
133
4.1
0.1
1.3
4,0
0.1
Stagel+2
19.8
6.6
1.8
8.6
20.2
3.0
11.1
21.5
8.2
62
11.8
13.8
Connector
Addition
1.6
18.0
1.8
11.2
11.5
21.9
Fixed Weights
3.1
9.8
19.4
5.6
2.4
9.3
20.7
7.9
4.3
11.8
26.1
15.3
7.4
12.7
30.6
36.7
Frame Number
TriSense (7B)
3.2
19.9
9.3
19.5
11.1
29.8
29.2
TriSense (78)
64
3,4
10.1
20.1
8.3
3.0
10.0
22.2
11.8
5.3
12.2
26.3
15.4
7.3
12.6
30.7
36.3
TriSense (7B)
128
3,4
10.2
20.2
8.5
3.1
9.9
22.8
11.5
5.4
12.3
26.7
15.4
12.8
30.8
36.1
We can observe that in Training Stages section, Stage focuses solely on modality alignment and temporal information is included, therefore does performing well across two tasks. However, after training in Stage 2, the model acquires around 50% of its capability. For the Connector ablation, we find that simply adding all modalities together does allow the model emphasize the more important modalities, resulting in a drop in performance. Similarly, in the Fixed Weights ablation for AVS/VS/AV tasks, assigning equal weights to modalities fails effectively capture their varying importance, leading to inferior performance compared to using dynamic modality weights. However, fixing visual weight indeed leads to slightly better performance in visual-only (V) setting. We also observe that increasing the number of frames used by the model (from 64 to 128) leads to performance improvement every scenario. This trend is consistent with the findings in both of the Moment Retrieval and Segment Captioning.
6 Conclusion
In this work, we introduced TriSense, a novel multimodal large language model designed to ad- vance comprehensive video understanding by integrating visual, audio, and speech modalities. At the core of our model is the Query-Basc aidion-allowing the system to operate effectively across arbitrary combinations of input modalities. This capability is essential for real-world scenarios, where certain modalities may be partially avail- able or entirely absent. To support progress in this area, constructed TriSense-2M 2M, large-scale dataset containing over 2 million carefully curated samples. These samples span diverse scenes, durations, and modality alignments, offering rich foundation for training and evaluation. Through extensive experimentation, we demonstrated that TriSense consistently achieves state-of-the-ard performance on key temporal video understanding tasks, including moment retrieval and long-form video segment captioning. Our aadapiive framework marks a substantial step toward more flexible and human-like video understanding systems. It not only delivers strong performance in controlled evaluations but also shows robust applicability in real-world conditions with varying input configurations. We believe that both TriSense and the TriSense-2N 2M dataset will serve as valuable re- sources for future research in multimodal learning and temporal reasoning, enabling broader advances across a range of video understanding applications.
Connector, which enables dynamic, modality-adaptivy
Acknowledgements
This project is jointly supported by the University of Western Australia (UWA) HDR Scholarship, Australian Research Council ARC DP210101682, DP210102674, and the Australian Government through the Australian Research Council's DECRA funding scheme DE250100030. We thank Zhejiang Lab for providing the computational resources.
10

---

References
[1] Satanjeev Banerjee and Alon Lavie. Closer look at summarization evaluations. In Proceedings of the workshop empirical modeling of semantic equivalence and entailment, pages 1-8, 2005. 7
[2] Sihan Chen, Xingjian He, Longteng Guo, Xinxin Zhu, Weining Wang. Jinhui Tang, and Jing Liu. Valor: Aision-audio-languaee omni-perception pretraining model and dataset. arXiv preprint arXiv:3304008355
2023. 3, 4,5
[3] Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. Vast: vision-addio-uubiite--eext omni-modality foundation model and dataset. Advances in Newrol Information
Processing Systems, 36, 2024. 3, 4,5
[4] Sanyuan Chen, Yu Wu, Chengyi Wang. Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. Beats:
Audio pre-training with acoustic tokenizers. 2022. 5, 14
[5] Zesen Cheng, Sicong Leng, Hang Zhang. Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang. Ziyang Luo, Deli Zhao, et al. Videollama Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv preprint arXiv:2406.0744
2024. 3
[6]
Konstantinos Drossos, Samuel Lipping. and Tuomas Virtanen. Clotho: An audio captioning dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP),
pages 736-740. IEEE, 2020. 13, 14
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Imms in video analysis. arXiv preprint TrXXi:2405.21075 2024. 17. 18
[8] Jiyang Gao, Chen Sun. Zhenheng Yang, and Ram Nevatia. Tall: Temporal activity localization via language query. In Proceedings of the IEEE international conference on computer vision. pages 5267-5275 2017.
2. 3, 5, 7.9
[9] Tiantian Geng. Teng Wang. Jinming Duan, Runmin Cong, and Feng Zheng. Dense-localizing audio-visual events in untrimmed videos: A large-scale benchmark and baseline. Proceedings of the IEEE/CVF
Conference on Computer Vision and Pattern Recognition, pages 22942-2295 2023. 5
[10] Tiantian Geng. Jinrui Zhang, Qingni Wang. Teng Wang. Jinming Duan, and Feng Zheng. Longvale: Vision-uddo--ie language-event eveent benchmark towards time-aware omni moni-modal perception of long videos. arXiv
Vision-uudoo-aanuuaeeeooo
preprint Xiv:2241 I. 19772, 2024. 2, 3, 4, 5, 7,8,9 9
[11] Yongxin Guo. Jingyu Liu. Mingda Li. Xiaoying Tang, Qingbin Liu, and Xi Chen. Trace: Temporal grounding video Ilm via causal event modeling, 2024. 2, 3, 5, 7,8, 8, 14, 15, 16, 17
[12] Yuchen Guo. Linchao Liu, Xin Li, and Ping Luo. Vtg-Ilm: Efficient temporal grounding in long videos with compressed visual cues. In arXiv preprint arXiv:2401.07683 2024. 2, 3, 6. 7, 16
[13] Fabian Caba Heilbron and Juan Carlos Niebles. Activitynet captions: A dense-captioning dataset for evaluating understanding of complex video activities. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5246-5255 2015. 5, 7,9
[14] Lisa Anne Hendricks, Oliver Wang. Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference
computer vision, pages 5803-5812, 2, 2017. 2,
[15] Junjie Huang, Ming Wu, Linchao Li, Yi Zhu, Yu Chen, Siwei Yan, Yi Liu, and Ping Luo. Vtimellm: Compression of time into a latent embedding for efficient video-language modeling. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 16694-16704, 2023. 2, 3, 7, 8 [16] Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Hel- yar, Aleksander Madry, Alex Beutel, Alex Camey, et al. Openai ol system card. arXiv preprint a4Xiv:2412. 16720, 2024, 4, 15
[17] Alhert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux. Pierre Stock. Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and
William El Sayed. Mistral 7b, 2023. 14
[18] Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference computer vision, pages 706-715, 2017. 3 [19] Jiabo Lei, Licheng Yu, Mohit Bansal, and Tamara I. Berg. Tallyqa: Answering complex counting questions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 14168-14178, 2021. 2
[20] Chin- Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd annual meeting of the association for computational linguistics (ACL-04). pages 605-612, 2004. 7
[21] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023. 13, 14
[22] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing 7hu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318, 2002. 7
11

---

[23]
Yiyuan Qian, Linchao Liu, Xin Li, and Ping Luo. Momentor: Advancing video understanding with temporal reasoning in large language models. In arXiv preprint arXiv:2401.0392 2024. 2, 3,7
[24] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PmLR,
2021. 5
[25] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PmLR,
2021. 14
[26]
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision, 2022. 5,
(27] Junjie Ren, Can Li, Ming Zhao, Jinhui Liu, Junchi Yang, and Jian Wang. Timechat: A time-aware large language model for vidco question answering. In Proceedings of the IEEE/CVF Conference Computer
Vision and Pottern Recognition, pages 16726-16736, 2023, 3.7
[28] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Vidcoclip: Contrastive pre-training for zero-shot video-text understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16632-16642, 2022. 2
[29] Cassia Valentini-Botinnn
et al. Noisy speech database for training speech enhancement algorithms and tts models. University of Edinburgh. School of Informatics. Centre for Speech Technology Research (CSTR), 2017. 13, 14
[30] Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh, Cider: Consensus based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[31] Hao Wang, Linchao Liu, Xin Li, and Ping Luo. Hawkeye: Visually explainable reasoning in video question
answering. In arXiv preprint arXiv:2401. 04405, 2024. 7
[32] Yi Wang. Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan Hc, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan
Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding.
arXiv preprint arXiv:2403. 15377, 2024. 3,4
[33] Yongliang Wu, Xinting Hu, Yuyang Sun, Yizhou Zhou, Wenbo Zhu, Fengyun Rao, Bernt Schiele, and Xu Yang. Number it: Temporal grounding videos like flipping manga. arXiv preprint arXiv:2411.1083 2024.
[34] Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang. Xiong Wang. Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report. arXiv preprint arXiv:2503.2022 20215, 15, 2025.
2, 7, 8
[35] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mci Li. Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin. Tianhao Li, Tingyu Xia. Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang,
Wan. Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5
technical report. arXiv preprint arXiv:24/2.1511 15, 2024. 4
[36] Hang Zhang. Xin Li. and Lidong Bing. Video-Ilamal An instruction tuned audio visual language model for video understanding. arXiv preprint arXivv2.306.022.6
2023.
[37] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data, 2024. 13. 14
[38] Bolei Zhou, Yu Guo, Meng Zhang. Xiongwei Wang. Siyuan Pu, Yuan Wu, Yan Zhang, Yan Wang. and
Li Li. Recent advances in video understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023. 2
12

---

Contents of Appendix
Implementation Details
13
A.1
Training Recipe
13
A.2 Datasets
13
A.3
Detailed training settings
14
B More details of TriSense-2M
15
B.1
Training Generator and Judger
15
B.2
Training data format
15
C Additional Experiments.
15
C.I
More ablation studies
15
C.2 General understanding dataset
17
Case studies on diferent scenarios
18
A
Implementation Details
A.1 Training Recipe
Our training process follows a structured three-stage approach: Feature Alignment, Connector Generalization, and Instruction Tuning progressively equip the model with strong multimodal and temporal reasoning capabilities. The specific components trained at each stage are illustrated in Figure 4.
Feature Alignment. the first stage, only the Query-Based Connector and LM Head are set as trainable, while all other components remain frozen. This phase employs single-oddlly, inputs, enabling the model to gain an initial understanding each of the three modalities individually. also helps the model learn assign weights effectively in the absence of multimodal context, promoting focused grasp of mopelity-specific features without interference from other components.
Connector Generalization. During the second stage, we incorporate mixed-modalin data and allow training for the Query-Bate Based Connector, Time Encoder, Time Head, and LM Head, while keeping the LLM backbone fixed. This phase equips the connector to handle weight allocation across multiple modalities, thereby enhancing its generalization beyond isolated modalities. Simultaneously, training the Time Encoder and Time Head introduces the model to temporal structure, laying the groundwork for capturing inter-modality dynamics over time.
Instruction Tuning. In the final stage, we freeze the Query-Based Connector and train the remaining components-inclludee
the Time Encoder, Time Head, LM Head, and the LLM backbone- uuing mixed- modelity inputs. By keeping the connector fixed, we retain the modality alignment learned in previous stages. This step concentrates on refining temporal reasoning and language understanding capabilities, strengthening the LLM's ability to interpret and process multimodal, temporally-sensitin queries across diverse scenarios.
A.2 Datasets
In alignment with the objectives of the three-stage training process outlined earlier, we employ varying datasets and data volumes each stage. The overarching goal is enhance the model's capacity for temporal video understanding while retaining robust general video understanding abilities. An overview of the datasets used in each stage provided in Table 9.
Table 9: Datasets and sample sizes used across the three training stages.
Stages
Datasets
Total Quantity
Stage
Clotho [6], L. [21], Salentini-Botinhan Speech Datasel [29]
600K
Stage
TriSense-2M (880K), LLaVA VALLVA-- 178K (120K) [37]
Stage
TriSense-2M (1.12M), LLaVA -ido--- (380K) [37]
1.5M
13

---

Stage1. For the initial stage, we use combination of the Clotho [6], LLaVA-LCS558K [21], and Allentini-Botinhar Speech Dataset [29] as training dataset:
Clotho is an audio captioning dataset containing 4,981 audio clips, each paired with five unique captions, totaling 24,905 annotations. The audio clips range from 15 to 30 seconds, and each caption consists of 8 to 20 words.
LLa VALCC5588 is concept- concept-balanced multimodal dataset comprising 558, 3,,000 image-texi pairs, annotated using BLIP- -generreed captions. is designed support feature alignment during the pretraining of vision-
models.
Botiiinn--otiihho Speech Dataset a parallel corpus of clean and noisy speech record- ings. It is widely used in training and evaluating speech enhancement and text-to-speech (TTS) systems, featuring 48kHz audio from multiple speakers under various noise condi- tions.
Stage 2 and 3. For these stages, we adopt our newly proposed TriSense-2M dataset, applying a 9:1 training-tor training-to-tetting to-testing split. This results in 1.9 million training samples and 0.1 million test samples. The training data is further partitioned into approximately 880K samples for Stage 2 and 1.12M samples for Stage 3.
To ensure the model also retains general video understanding capabilities, we supplement the training data with a portion of LLLVVA-Vdeo- 17KK [37], which includes video captioning, open-ended QA, and multiple-choice QA tasks. This mixed-tass dataset helps the model develop broader understanding skills beyond temporal reasoning.
Table 10: Details of test set.
Num. Videos
Num. Events
Avg. Event Duration
Total Tasks
3805
11415
7.23s
91320
To avoid massive evaluation time, we extract 11,415 challenging samples from the IM test set, as shown in Table 10 using two filtering criteria: 1) The majority of events should occur in the middle portion of the video rather than the beginning. 2) Captions must contain at least 20 words. Evaluation is conducted using single A100 SXM4 80GB GPU with batch size of 1, requiring approximately 8-10 hours to complete. All models in comparison are evaluated on this same test subset, using their officially recommended hyperparameters (e.g., number of frames, temperature, top-p, etc.)
A.3 Detailed training settings
Our multimodal framework incorporates dedicated encoders for each modality. For the visual modality, we adopt
[25]; for audio and speech modalities, we employ BEATs iter3+ (AS2M) (cpt2) [4] and Whisper-large-V. V3 [26], respectively. As for the large language model (LLM) backbone, select Mistral-7B [17], initialized from TRACE [11], instead of using other LLM backbones, rather than using other LLM backbones. This choice is motivated by TRACE's prior training large-scale temporal understanding data, which equips with stronger temporal reasoning capabilities. The maximum context length is configured to 4096 tokens.
During training, videos are resampled at 1 frame per second (fps) to improve efficiency shficiencyythis omitted during inference to retain full fidelity. This resampling reduces input redundancy and accelerates training.
this step
In Stage 1, we train the model with a batch size of 512 and single-frame input, completing within 10 hours using x A0000004-08000
GPUs. Stage 2 and 3 are conducted on 16x A 100SXM4-80GB GPUs with batch sizes of 128 and 256, respecitvely. Stage requires approximately 3.5 days to complete, and stage 3 requires 7 days to finish. We use DecpSpeed Zero2 because the BEATs model does not function properly under Zero3 settings. Further details on datasets and hyperparameters are provided in 11.
14

---

Table 11: Training configurations and hyperparameters by stage.
Settings
Stage
Stage 2 & Stage3
Computation
4x 4xA100SXM4-00GB.
16x A A100SXM4-80GB
Vision Encoder
hip-vit-large-puccl4--33
hop-vi--lagg--aacch---33
Audio Encoder
BEATs iter3+ (AS2M) (cpl2)
BEATs iter3+ (AS2M) (cpt2)
Speech Encoder
Whisper-large- V3
Whisper-large- Zero2 Offload
DeepSpeed Stage
Zero2 Offload
LLM Backbone
Mistral-7B-v0.2
Mistral-7B-vO..
Batch Size
512
128 & 256
Num Frames
1
64
Frame Sample
Uniform
Uniform
Train Epochs
2
Learning Rate
lc-3
5c-6 Cosine 4096
LR Scheduler
Cosine
Model Max Length
4096
Training Duration
10 Hours
3.5 days & 5.5 days
B More details of TriSense-2M
B.1 Training Generator and Judger
This section describes the data generation and manual filtering process used to prepare training data for both the Generator and the Judger. To ensure efficiency and quality in omni-modal caption generation, we first utilize GPTol [16] to produce high-quality annotation samples for Supervised Fine-tuning (SFT). The specific prompts used for this process shown in 5. These prompts serve dual purpose: they are used to generate training data via GPT and also function as system prompts during the SFT of both the Generator and the Judger.
To further enhance caption quality, we implement a two-stage scoring mechanism. After captions are generated, GPT conducts self-evaluation. Then, separate GPT instance provides an additional evaluation to filter out low-quality samples. Following this automated scoring, we conduct manual sampling verify consistency and ensure a high quality standard is met.
Data is generated batches of 1,000 samples. From each batch, we randomly select 500 samples for manual review to evaluate the generated content and the reliability of GPT's scoring. If over 80% of the reviewed samples meet our quality criteria, the batch is retained; otherwise, it is discarded.
Ultimately, we curate 10,000 training samples for the Generator and 3,000 samples for the Judger. Both models trained for epochs to establish effective captioning and judging capabilities.
B.2 Training data format
This section outlines the data format used for training TriSense. We adopt a ShareGPT-style format, where each training sample consists of 8 conversation rounds, each corresponding to different modality combination. The tasks and settings within these rounds are randomized for instance, one round might involve a VS-SC task, while the next could be AVS-SC or V-MR task.
Following the approach in TRACE, we use special tokens such as (sync) and (time) to signal the model to switch between different prediction heads. An example of the data structure is illustrated 6.
Additional Experiments.
C.1 More ablation studies
We conduct small-scale ablation studies on slot compression and task-specific heads. Specifically, we randomly selected 20K samples and trained for 3 epochs, initializing all weights from [11]. All experiments were conducted with a sampled frame rate 64. In terms of token compression, we explored different slot configurations, including 8, 32, and 64. For the prediction head, removed
15

---

helpful assistant designed to oulput CAPTIONS ISON. Given three distlnct caplions describing audio, visual, and speech scenarios, please generate omni-modal captions for the following combinations: Ludio-Visaal-Spee Seeeal--Seehhhhce (AVS). Visual-Speech (VS). Audio-Visual (AV).
You a helpful assisiant designed to evaluate coption quality. Given three original caplions and three generated omni-modal caplions, please assess the quality of the generated caplions based on the following citteria:
generated captians must include key information from the orgginalcppiion captions,
Input Examples:
Audo: (audic) (audic'_caation)
2. No new content may be introduced beyond what is present in the original caplions.
Visual: vijudl'_copliir)
Speech: (speechl_capp cappioo)
generated captions must not copy text directly from th original captions; instead, they should paraphrase and incorporate the essenticl information.
You should generate captions according the following rules:
The genercted capliens must preserve essential information from the original caplions.
Eoch generated caption
exceed words.
2. Do not infroduce any content that & not present in the original captions.
Input Examples:
Audio: (audio coption)
copy verbofim
original caplions: inslead.
ViJucl: MisuaLcoplion
paraphrase the key information and incorporate it into the new caption.
AVS: (audio-viuull-g] visucl-peechh caption)
Soeech: (speech, caption)
VS: (vSuol-speech _coplioon
Each genercted capton must exceed 200 words.
(oudio-itudl, _ctppiinccooonn
When you generate captions, assign a score for yourself from to It the you think the generated captions satisfy approximately 80% the above criteria, assign a score assign a score of 5.
Assign a score from 1 to 5. the quality very poor, assign score of I. the generated captions salisfy approximately 80% of the above criteria, assign score (100%). assign a score of
assign score of 1. If the
they fully salisty all criteria
they fully satisly criteria (100%).
Please analyze carefully and provide evalvation in the following JSON format: ("AVS": " "AV": "V5". "Score": "Nooe that you should provide only one score and one caption for each of AVS, AV, and VS. You MUST respond In JSON format.
think carefully and provide your follows: ("AVS", "AV":" "V5":" "Score": "). Note that only one caplion should be provided for each of AVS, AV. and VS. You MUST respond JSON format.
format
Figure 5: Prompts used for training the Generator and Judger. The left prompt guides GPT in generating omni-modal captions for the Generator using audio, visual, and speech inputs. The right prompt is used to train the Judger by instructing GPT to assess the quality of generated captions based on coverage, accuracy, and paraphrasing. During data creation, samples are randomly selected and manually filtered to ensure high-quality training data.
the Time Hcad and instead encoded temporal information into special tokens, which were injected into the LLM. Prediction was then performed using only the LM Head.
Table 12: Ablation study on the slot compression [12].
AVS-SC
AVS-MR
Avg. Event Duration
Slot=8
B: 1.6 M: 3.1 R: 13.1 C: 1.9
1oU=0.5: 0.3 IoU=0.7: 0.1
Slot=16
B: 1.8 M: 3.4 R: 14.6 2.4
IoU=0.5: 0.4 IoU=0.7: 0.3
Slot=32
B: 1.8 M: 3.4 15.6 C: 2.4
1oU=0.5: 0.4 1oU=0.7: 0.3
Slot=64
B: 1.9 M: 3.5 R: 16.1 C.2.4
1oU=0.5: 0.5 1oU=0.7: 0.2
Table 13: Ablation study on the task-specific heads.
AVS-SC
AVS-MR
Unified Head
B: 1.5 M: 3.1 R: 13.2 C: 1.1
1oU=0.5: 0.2 1oU=0.7: 0.1
Time Head LM Head
B: 1.8 M: 3.4 R: 14.6 C: 2.4
1oU=0.5: 0.4 1oU=0.7: 0.3
As shown in Table 12, for slot compression, although reducing the compression rate (by increasing the number of slots) can bring slight performance improvements, it also results in significant increase in computational overhead. Therefore, the benefits are subject to diminishing returns. As shown in the table, setting the slot to 16 currently offers good balance between performance and computational cost. For task-specific head in Table 13, using a single unified head alone leads to a significant drop in performance, indicating that the time head provides additional temporal information which is beneficial to the model. There may be better unified solutions in the future.
We also provide more ablation studies in Table 14 and Table 15 for different branches of the Query- Based Connector based on the same 20K randomly selected samples as above. For example, AV-Only in the table means we discarded the Speech branch. During training, all the weigths are initialized from [1I], and LLM backbone is set to frozen. Results show that all three modalities contribute positively to all tasks, with the full AVS configuration performing best.
16

---

Dalasef Format Example of ShareGPT
"video": "EegicUE280.mpp
"conversations".
'from": "human".
value": "viceo>\/nfiom 1070.8 to 1081.9, please detail the sound event, highighlligg any essential visual and
Trom': "gpt".
volue":
passionately speaking on stage while the audience laughs and talks in the theater. The crowd engoged and responsive."
Trom':
"human":
volue": "Could you provicle a summary of the video between (1070.8, 1081.9). locusing on visuol, audio, and speech elements
Trom": "gpf".
value": <sync=<limee<<imme->>
person performing stage, oudience loughing talking in theater, speech subtitle that the person has to hide because they look like rot from the movie Ratatouile.."
'rrm": "human".
"value":
find the precise time seconas when the content described os la man is singing takes place the video, and report timestomp."
from": "gpf".
singing on stage"
<Remaining conversations....
"Imess: [[1070.8, 1081.9), (1070.8. 1081.9), [1070.8. 1081.9), <remaining times.......
Figure 6: Example of ShareGPT-stylle annotation format used during training. Each sample includes multi-turn conversations video segments with synchronized modality cues. Only the first three rounds are shown due to space limitations.
Table 14: Ablation studies on the branches of Query-based Connector on Segment Captioning tasks.
Model
AVS-SC
VS-SC
AV-SC
V-SC
V-Only
B: 1.0 M: 1.7 R: 10.1 0.9
B: 1.1 M: 16R: 10.3 1.1
B: M.0 1.7 R: 10.2 C: 1.1
B: 1.0: R 13.3 1.4
VS-Only
B: 1.7 M: 2.9 R: 13.3 C: 24
29R 14.7 2.5
B: 0.7 M: 2.3 R:
13.2 1.5
B:0.7M: R 13.2
AV-Only
3.1
12.0:2211
1.7
AVS Full
B: M: 3.4 14.6 2.4
1.7 1.7 15.1 2:2.5
B: 1:1.0.:: 3.2 15.3
R:0.M:: 13.4 1.5
C.2
General understanding dataset
For general video understanding evaluation, we report results on VideoMME [7]-a large-scale benchmark designed assess multimodal large language models (MLLMs) in video analysis. VideoMME covers wide range of visual domains, temporal scales, and modalities, including 900 videos (totaling 254 hours) and 2,700 numan-annoter not only demonstrates significant ad advantages in multimodal scenarios but also performs competitively in general understanding tasks. It is worth noting that our model uses much less general understanding data (500K) compared to TRACE-uni [11], which uses 0.9M, as reported in their official paper.
QA pairs. As shown Table 16, our model
17

---

Table 15: Ablation studies the branches of Query-based Connector Moment Retrieval tasks.
Model
AVS-MR
VS-MR
AV-MR
V-MR
V-Only
1oU=0.5: 0.1 1oU=0.7: 0.0
IoU=0.5: 0.1 loU=0.7: 0.0
IoU=0.3: 0,1 1oU=007: 0.1
1oU=0.5: 0.4 1oU=0,7: 0.2
VS-Only
1oU+0.5; 0.4 1oU:0.7: 0.1
1oU=0,5; 0,3 1oU=0.7: 0.2
1oU=0.5: 0,1 1oU=007: 0.0
1oUs0,5: 0.3 1oU=0.7: 0.1
AV-Only
1oU=005: 0.4 1oU=0.7: 0.2
1oU==05: 0.1 loU=0.77 0.1
1oU)=0.5: 0.3 1oU=0.7: 0.1
loU:005: 0.3 1oU=0.7: 0.1
AVS-Fall
loU=0.5: loU=0.7: 0.3
1oU=0.5: 0.3 loU=0.7:
IoU=0.5: 0.3 IoU=0.7: 0.2
1oU=0.5: loU=0.7: 0.2
Table 16: Zero-shot performance on general understanding dataset Video-MME [7]. Our model only uses 55% of the general understanding data compared TRACE-uni.
Model
VideoMME (Overall Scores w/o Subtitles)
VideoChat2 (7B)
33.7
Video- (7B)
39.9
VideoL.LaMA2 (7B)
46.6
TRACE (7B)
43.8
TRACE-uni (7B)
49.6
TriSense (7B)
48.7
D
Case studies on diferent scenarios
We provide case studies of TriSense in various scenarios in Figures 12.
AVS-SC
Sound: Possionaae and upifting music.
FIFA17 URTIAIS
559.5
567.65
700s
Speech: Yes we managed get it 3-1, absollueely horific finis! But DeCosta get
loday's game, ployer
time frame 667.4) please describe oudio-vtuull-peect event vhiblle cilp.
Ground Truth: boy is playing video game and talkding about and player nomed DeCosta scored his second goa of the game. iu was a lough fink..
weiting
black shint, standi clurte enthuslasfcclly diring
instill new set fire on trusk.
LongVVLL
0
part video. It's a guy talking, daing anything visully Hels just specting. you about
Qwern..5-Omri
Timestames": (1559.5 557 "capfions", the bockground, and the speaker mentions
football
while
3
Trisense
Madalily Weights: Audiovd 1854, Speech=0, 1038, Vision00.7108
Figure Case study of TriSense on AVS-SC task.
18

---

V-SC
Sound: background sound/music
Os
10s
475.2s
477.45
700s
710s
Speech: Cut around the stane...
range (475.2, 4, please video's content, focusing shiiilyy
diing curtingg
Humon
o LongVALE
A mon caretully stiing pol of thick, reddish brown souce on the stove, as he expicine the process making tomats souce.
Al that time. there's a text that VON THE NDTF NOTICIATION SQUADIY the screan, It's proballly encouragement viewers to subsorfoe and notifcattons is the chonnel
Owen225-Omri
0
"innstamp"" [[75.22 47) "caplions", I'' man culling piece ovocado on culling beard
TriSense (Ours)
Modality Weight:: Audior0: Speech-=,0077 Mslon=0.8223
Figure 8: Case study of TriSense V-SC task.
AVS-MR
Sound: Rhythmic and dynomic music.
216.3s
223.45
250s
2555
can capture II frames a second with the mechanical shufter. And up to frames a second with the electronic shufter.
Speech:
second. capture
Querr:: lacaze the moment where 'the man jumping in the ali, with the speaker soying that mechanical shutter is sei frames per second, while possionate music playing the background. The speoker then soys up flomes per second, mechanical shufter and electronic Shulfer.)", r.fumm precse tir tirestopp.
Human
Ground (216.3. 223
From 05 to 24
LongVALE
(200, 2001
Qwen2.5 Omni
"imestamp" [216&, 223 Bj] "captons") The man jumoing in the ai with the speaker soying that mechanical shurter set per socondl possionate music ploying bockground. The speaker Jays frames par second. and elactroric shutterA.1
(Ours)
Vislonw0.7091
Figure 9; Case study of TriSense on AVS-MR task.
19

---

VS-MR
Sound: slent background sound
0s
10s
189.3s
197.95
980s
Speech: This one B aso shipped to us but we knew the status of Ihis Abino. would not have shipped.
Query: Determine precise seconds when person coge.
showing large yellow
crowling
enclosure people hoppens
Humon
Ground Truth: (199.)
LongVALE
The content
video 1800 240.0
@weri2.5-Omi
0
"Imestamps": [[1882, 1978. "capllons: -A person describes video about on Abino snake while showing large yellow dike inside cage. The snoke crowling cround in enclocure people tolk.]
TriSense (Ours)
Madalily Weights: Audioso, 1035,
Speach=0, 1842, Vision00.7123
Figure 10: Case study of TriSense VS-MR task.
AV-MR
Sound: Crocking sound coused by o screworiver smoshing phone
LETT
0s
10s
659.75
666.15
980s
Speech: This our Android wish phone. want to see these [phones) are alive, Chiil It started smoking...
Query:
moment deo coresporde query speaking into microphone. makinn
discussing content presented!"
laughing. spectic
Humon
Ground Truth: (659
0
LongVALE
(0000
Qwen225-0mmi
0
"Imestamps" [[688.1. 645.0]. "captions' l'A man studio speaking nto a microphone, decuszing the features of electronic device while mak ng sounds with his hand and loughing.1
TriSense (Ours)
Modality Weights:
Audion0.2674 Speech=6..22 0238, Visione0, 7088
Figure 11: Case study of TriSense on AV-MR task.
20

---

General Understanding
05
10s
60s
70s
womon holding (0.0. 5.0]?
cen
Humon
Please
phane
stating the followed by
'imeslorpp
(' paper ']
Treerse
Modal Audion0: Speechno 0485, Vidion007666
What holding hand between
Bio pekce poper
C.o book
Human
Please provide your answer by stating the letter tollowed by the full option.
D.d phore
"meetorpp:: 'captions: pen
0
Medal Audio=0.2004 Speech#0 0502, Vision=0.7455
Träerse
Figure 12: Case study of TriSense on General Understanding task.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have adequately addressed claims in Abstract and Introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
Due to the page limit, we include the limitation discussion in appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
22

---

Justification:
We have properly cited relevant papers and have doubled checked theories.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, the code and data will also be made public.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
23

---

Answer: [Yes]
The code an data will be made public on cloud drive and git repository after review.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nipe cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https:
wnips.cc/ppi wttpcbcccppccccccccccccccccccccc guwdee/deeeeeee Cooeeubmiis www.onooliiieoooooooooooooeoeaee for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The details have been included in experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The paper does not include eπror bars due to the large scale of the experiments (More than 10,000 A100 GPU Hours), resources limited do so.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Training resources have been included in imlementation details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt///neuripppp wttpp/cccc/tpc...ceoooooocccctle ceuihe/deee innes???
Answer: [Yes]
Justification:
We have checked and followed the ethical guidelines.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
25

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper poses no such risks, and the usage safeguards will be included in git repository by proper license.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have checked all the licenses of the used datasets in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
26

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We do introduce new assets in this paper, have included details in the paper and appendix.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The LLM is only used for writing, and the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: //tprr/ss...///// cc/Conf / /LLM) for what should or should be described.
28


Reviews:
[{"review_id": "YoBpeObh1c", "reviewer": "Reviewer_ThZF", "summary": "This paper introduces TriSense, a multimodal large language model architecture designed for comprehensive video temporal understanding, integrating visual, audio, and speech modalities. The core novelty is a Query-Based Connector that adaptively weights and fuses modalities based on the input query, enhancing robustness to missing or irrelevant modalities. The authors also present TriSense-2M, a curated, large-scale dataset of over 2 million annotated long-form video segments incorporating variable modality combinations. Extensive evaluations across segment captioning and moment retrieval tasks on both TriSense-2M and several public benchmarks show that TriSense outperforms prior models, notably excelling in settings with partial modality availability. Ablation studies further substantiate the contributions of the adaptive connector and dataset design.", "questions": "My main concern is the novelty of the modeling, as I do not see substantial innovation in the design. But I appreciate the introduction of a high-quality AVS dataset, which is a valuable contribution to the community. Overall, I prefer to be borderline; but, since a strictly neutral rating is not available, I lean slightly positive and therefore give a borderline accept. I will make my final decision after reading the authors’ rebuttal.", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n* The paper is clearly written and easy to follow, with well-organized sections and clear explanations of both the method and experiments.\n* The idea of a Query Based Connector to dynamically reweight contributions from different modalities is intuitive.\n* The introduction of the new TriSense 2M dataset is a valuable contribution to the community.\n* The proposed method achieves significant performance gains over strong baselines across multiple benchmarks, demonstrating its practical effectiveness.\n\n Weaknesses\n* Limited novelty in modality fusion techniques. While the Query Based Connector is positioned as the key modeling innovation of the paper, its design closely resembles existing approachesp[1,2]. Specifically, the query based cross attention mechanism is highly similar to the instruction-aware QFormer proposed in InstructBLIP and X-InstructBLIP[1]. In addition, the dynamic reweighting of modalities resembles the self-gated query fusion strategy used in CREMA[2]. As such, the proposed connector appears to be more of a combination of these two prior techniques rather than a fundamentally novel modality fusion method. This overlap limits the originality of the technical contribution.\n* Missing citations and related work discussion. Given that the Query Based Connector is the central technical innovation of the paper and even emphasized in the abstract, the lack of a thorough discussion of related work on modality fusion is a notable weakness. A more comprehensive related work section discussing these and clarifying how the proposed connector differs or improves would strengthen the paper and help position it more clearly in the landscape of multi modal fusion research.\n\nReferences\n\\[1] X InstructBLIP: A Framework for Aligning Image 3D Audio Video to LLMs and its Emergent Cross modal Reasoning 2023\n\\[2] CREMA: Generalizable and Efficient Video Language Reasoning via Multimodal Modular Fusion ICLR 2025", "was_revised": true, "final_justification": "The author's rebuttal on novelty is clear. Initially, I felt the work was borderline but leaned toward a borderline accept due to the lack of a neutral rating option. With the clarification provided, I now believe the current score is appropriate and will maintain my positive rating.", "rebuttal": "**First of all, we would like to sincerely thank the four reviewers for their constructive feedback on our manuscript:** \"The paper is clearly written and easy to follow, with well-organized sections and clear explanations of both the method and experiments\", \"The introduction of the new TriSense 2M dataset is a valuable contribution to the community\" (Reviewer ThZF); \"This paper studies a very interesting problem which seeks to build an omni-modal large language model for comprehensive multi-modal understanding\" (Reviewer 1DNL); \"The authors propose a groundbreaking adaptive weighting framework validated by TriSense-2M\" (Reviewer NaTe); \"Make substantial data contributions, which are significant for omni-model exploration\" (Reviewer 9tN8). **Next, we will carefully address each of the reviewers’ comments.**\n\n> Q1: Limited novelty in modality fusion techniques. While the Query Based Connector is positioned as the key modeling innovation of the paper, its design closely resembles existing approaches[1,2]. Specifically, the query based cross attention mechanism is highly similar to the instruction-aware QFormer proposed in InstructBLIP and X-InstructBLIP[1]. In addition, the dynamic reweighting of modalities resembles the self-gated query fusion strategy used in CREMA[2].\n\nWe would like to respectfully claim that the proposed Query-Based Connector **is NOT** the combination of these techniques, with detailed explanations below:\n\n- **Architectural differences:**\n  1. **QFormer [1,2,3,4]** uses **fixed, learnable query vectors** to perform cross-attention across modalities, generating instruction-aware, fixed-length representations. In contrast, our method uses the **actual user-input prompt** as the query, making it flexible and **only used in the initial stage.** Because the cross-attention facilitates cross-modal interaction, placing the query alongside the target modality is a **natural and intuitive design choice.** Therefore, despite similarities, our approach **differs significantly** from QFormer in both the nature of the query and our overall objectives, as cross-attention is not the central contribution of our work.\n  2. **Self-gated Multimodal Query Fusion [4]:** In CREMA, video is treated as the **primary modality**, while other modalities are linearly compressed as **supportive modalities**, gated using a sigmoid function, and fused into a fixed-length token alongside the primary modality. In contrast, our Query-Based Connector treats all modalities equally, without designating any as primary or supportive, and **does not aim to align them to the primary modality.** Therefore, our proposed Query-Based Connector is **totally different** from Self-gated Multimodal Query Fusion from both architecture and concept.\n  3. **Proposed Query-Based Connector:** The query attends to each modality (visual, audio, and speech) independently via cross-attention to extract query-relevant features, this is only an **initial step** that is similar to QFormer, but **only superficially.** Next, all modalities are processed through an MLP followed by a softmax layer to generate dynamic weights conditioned on the query. This enables adaptive modality reweighting and temporally aligned fusion, which naturally suppresses missing modalities. **Unlike QFormer, our method goes beyond basic query-to-modality cross-attention. Unlike CREMA, we do not designate a major modality or attempt to merge other modalities into the major one. Notably, none of the referenced works incorporate modality reweighting.**\n- **Design Intentions Difference:** The primary goal of QFormer is to extract **single-modality** features to mitigate the modality gap. In contrast, CREMA’s self-gated Multimodal Query Fusion [4] is designed to **prevent the query token size from scaling linearly** with the number of modalities. Our Query-Based Connector, however, is built with a different design philosophy: to offer a **universal, simple, and effective** mechanism for extracting features from all modalities while enabling **dynamic weighting and fusion.** This approach naturally favors a concise connector architecture to maintain generalizability, avoid overfitting, and reduce optimization challenges\n\nLastly, we want to respectfully emphasize that the connector itself is **carefully designed,** incorporating design elements such as inter-frame and inter-token positional encoding, the placement of LayerNorm, and the adjustment of softmax temperature—all of which contribute to its effectiveness in omni-modal settings. Our key contribution lies in demonstrating that such a concise and adaptable architecture can perform well across diverse modalities, a capability that, to our knowledge, **has not been shown in prior VLMs.** We believe this offers meaningful insights for future research. The proposed TriSense-2M dataset is intended to serve as a **robust pretraining foundation** for large multimodal models. Additionally, the TriSense model trained on this dataset can act as a **strong foundation model** for future studies, supporting fine-tuning and enhancement while substantially reducing training costs.\n\n**References:**\n\n[1] Dai, Wenliang, et al. \"Instructblip: Towards general-purpose vision-language models with instruction tuning.\" *Advances in neural information processing systems* 36 (2023): 49250-49267.\n\n[2] Panagopoulou, Artemis, et al. \"X-instructblip: A framework for aligning image, 3d, audio, video to llms and its emergent cross-modal reasoning.\" *European Conference on Computer Vision*. Cham: Springer Nature Switzerland, 2024.\n\n[3] Zhang, Hang, Xin Li, and Lidong Bing. \"Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding.\" *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations*. 2023.\n\n[4] Yu, Shoubin, Jaehong Yoon, and Mohit Bansal. \"CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion.\" *The Thirteenth International Conference on Learning Representations*. 2024.\n\n> Q2: Missing citations and related work discussion. Given that the Query Based Connector is the central technical innovation of the paper and even emphasized in the abstract, the lack of a thorough discussion of related work on modality fusion is a notable weakness. A more comprehensive related work section discussing these and clarifying how the proposed connector differs or improves would strengthen the paper and help position it more clearly in the landscape of multi modal fusion research.\n\nThe discussion of the Query-Based Connector in relation to existing connectors was concise in the original submission. We have now provided a detailed comparative analysis addressing this point in your **last concern** and will incorporate it into **future revisions** to further clarify the distinctions and strengthen the paper’s contribution."}, {"review_id": "oOZ4GnArq2", "reviewer": "Reviewer_1DNL", "summary": "This paper introduces a novel triple-modality large language model for detailed video understanding that considers visual, audio and speech modalities. The proposed model incorporates a query-based connector which can automatically fuse multi-modal features. Besides, the authors collect a large-scale dataset comprising 2 million samples with synchronized video, audio and speech. The experimental results and the downstream applications demonstrate the effectiveness of the method and dataset.", "questions": "1. Could you provide a detailed explanation of the feature shapes for the vision, audio, and speech modalities? Additionally, how many tokens are ultimately fed into the large language model?", "limitations": "No. In line 273, the authors find that “model shows slightly lower performance on visual-only moment retrieval compared to state-of-the-art vision models”, future work could explore methods for balancing the multi-modality question answering performances.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strength:**\n1. This paper studies a very interesting problem which seeks to build an omni-modal large language model for comprehensive multi-modal understanding. \n2. The proposed TriSense-2M dataset is useful and contains various types of captions including AVS, AV, VA captions, which can support a wide range of future studies. \n3. The proposed query-based connector in the TriSense model is novel, as it extends the previous Q-former-based connector to support three modalities: audio, visual, and speech for multi-modal tasks.\n\n**Weakness:**\n1. The overall quality of the proposed TriSense-2M dataset is not very clear. Could you provide some randomly picked examples from the dataset?\n2. According to the recommendation of the NeurIPS conferences, the authors should discuss the limitation of their proposed models and potential future improvements.\n3. As mentioned in Appendix B, in the dataset construction pipeline, only 10,000 and 3,000 samples are used to train the generator and the judger, respectively. This raises my concerns about their captioning and judging capabilities for reliable data annotation.", "was_revised": true, "final_justification": "The rebuttal well addressed my concerns and I remain positive about the paper.", "rebuttal": "**First of all, we would like to sincerely thank the four reviewers for their constructive feedback on our manuscript:** \"The paper is clearly written and easy to follow, with well-organized sections and clear explanations of both the method and experiments\", \"The introduction of the new TriSense 2M dataset is a valuable contribution to the community\" (Reviewer ThZF); \"This paper studies a very interesting problem which seeks to build an omni-modal large language model for comprehensive multi-modal understanding\" (Reviewer 1DNL); \"The authors propose a groundbreaking adaptive weighting framework validated by TriSense-2M\" (Reviewer NaTe); \"Make substantial data contributions, which are significant for omni-model exploration\" (Reviewer 9tN8). **Next, we will carefully address each of the reviewers’ comments.**\n\n> Q1: The overall quality of the proposed TriSense-2M dataset is not very clear. Could you provide some randomly picked examples from the dataset.\n\nDue to NeurIPS and OpenReview rebuttal policies, we are **unable** to include any visual samples or external links at this stage. However, we fully acknowledge the importance of illustrating dataset quality through representative examples, we have also included a certain number of image samples in **Appendix D** of the initial version of the manuscript. In the meanwhile, to address this, we provide qualitative examples and evaluations in our **response to Q3**, where we include a selection of text-based samples spanning multiple modality combinations (audio, visual, and speech). These include source captions, generated captions, human-written references, and judger scores. To further promote transparency and reproducibility, we will release the code and data on GitHub.\n\n> Q2: According to the recommendation of the NeurIPS conferences, the authors should discuss the limitation of their proposed models and potential future improvements.\n\nBelow, we outline the current limitations of our work and future directions for improvement from two perspectives:\n\n1. **Dataset Quality:** While extensive filtering reduced the dataset from 5M to 2M samples, full manual verification at this scale remains challenging. Moving forward, we anticipate further quality improvements through integration of stronger models such as GPT-o3 or Gemini 2.5 Pro. Additionally, selectively expanding the volume of human-reviewed samples and scaling the dataset to 5–10M entries without compromising quality are promising avenues that will enhance the dataset’s utility in both pretraining and transfer learning tasks.\n2. **Model Design (Query-Based Connector):**  Potential refinements include the adoption of 2D-RoPE in place of absolute positional encodings, the substitution of LayerNorm with RMSNorm for improved stability, and the incorporation of activation functions like SwiGLU, which have shown efficacy in large language models. Moreover, our ongoing work includes optimizing the dynamic modality weighting mechanism to further strengthen multimodal alignment and representation fidelity. We are already implementing these improvements and will open-source them on GitHub.\n\nLastly, while we acknowledge the limitations, we believe that **the contributions of this paper outweigh its limitations.** These limitations are not fundamental flaws but rather natural outcomes of working at the frontier of model and dataset development. We view them as opportunities that will enrich the future evolution. The proposed **TriSense-2M** dataset is expected to serve as a **strong pretraining foundation** for large multimodal models. Furthermore, the **TriSense** model trained on this dataset can act as a **powerful foundation model** for future research, supporting fine-tuning and further improvements while significantly reducing training costs. To promote reproducibility and encourage continued development, we will release the full codebase, including the training pipeline and connector implementation on GitHub.\n\n> Q3: As mentioned in Appendix B, in the dataset construction pipeline, only 10,000 and 3,000 samples are used to train the generator and the judger, respectively. This raises my concerns about their captioning and judging capabilities for reliable data annotation.\n\nThe 10,000 and 3,000 samples refer to manually reviewed sets, however, due to the multiple-caption-per-sample design (AVS, AV, VS), the actual review workload was **4–5× larger**, which is already a heavy workload for us. To overcome the potential bias, we applied a strict acceptance threshold (**80%**) and set a **low pass rate** for the judger model, resulting in **rigorous filtering.** Out of 5M initial samples, only 2M were retained. \n\nTo further validate annotation quality, we conducted a **human evaluation** on 100 randomly selected samples generated by the generator. Each was **scored** by a human annotator who **has not seen the model output** according to the criterias in Figure 5. Lastly, **91/100** samples are passed by human, yielding a **91%** acceptance rate. The results show that while human captions were often more fluent, model-generated captions showed **strong** semantic adequacy and scalability advantages. Due to text constraints, we cannot show all of them to you, but we do our best to show you some in the table below, and we will include more samples in the future version.\n\n| Source Caption                                               | Generated Caption                                            | Human Caption                                                | Judger Score | Human Rate |\n| ------------------------------------------------------------ | :----------------------------------------------------------- | :----------------------------------------------------------- | ------------ | ---------- |\n| **audio:** a person is opening and closing an umbrella. **visual:** outside a woman is showing how to decorate a christmas tree. **speech:** So I thought these would sit up on the desk too and just be really pretty and neutral. So all of that came from Kirklands. | **avs:** A woman shows how to decorate a Christmas tree outside while a voice of opening and closing an umbrella is heard, and the woman thinks the decorations will look nice on a desk and be neutral in color, all from Kirklands. **av:** A woman is outside decorating a Christmas tree, and a person is opening and closing an umbrella somewhere else. **vs:** A woman sits outside decorating a Christmas tree and showing art she has finished, mentioning how pretty and nice they would look. | **avs:** A woman is decorating a Christmas tree outdoors. While decorating, she introduces to the audience that these decorations are from Kirklands, which can bring a pretty and neutral visual effect. Meanwhile, someone in the background is opening and closing an umbrella. **av:** A woman is decorating a Christmas tree outdoors. At the same time, someone in the background is opening and closing an umbrella. **vs:** A woman is decorating a Christmas tree outdoors. While decorating, she introduces to the audience that the decorations are from Kirklands, which can bring a pretty and neutral visual effect. | 5            | 4          |\n| **audio:** someone is messing with a cup. **visual:** a woman is showing a drawer full of lipstick. **speech:** So those are my lip balms and then these are all lip glosses. I have a lip tar, so thats, not the little car. | **avs:** A female is showing of her drawer full of various kinds of lipstick, including lip balms and lip glosses, and she mentions that she has a lip tar, messing with a cup in the meantime. **av:** A woman is messing with a cup and showing her drawer, which is full of different kinds of lipstick. **vs:** A girl is talking about her makeup drawer, mentioning various lip balms, lip glosses, and other beauty products, including a lip tar. | **avs:** A girl is showing a drawer filled with different lipsticks in the video. While introducing the various types of her beauty products, she is also messing with a cup. **av:** A girl is showing a drawer filled with different lipsticks in the video, while messing with a cup. **vs:** A girl is showing a drawer filled with different lipsticks in the video, and at the same time, she is introducing the various types of her beauty products. | 4            | 4          |\n\n> Q4: Could you provide a detailed explanation of the feature shapes for the vision, audio, and speech modalities? Additionally, how many tokens are ultimately fed into the large language model?\n\nAs described in Section 4.1, we use slot compression to reduce the number of tokens for each modality to 16. If we set the number of frames to 64, then for the **vision, audio, and speech** modalities, the feature shape will be $64$ frames $\\times$ $16$ tokens of 4096 dimensions, i.e., [$64$, $16$, $4096$]. The **query** typically consists of 40–50 tokens, i.e., [$40$, $4096$]. After inputting these four types of features into the Query Based Connector, it outputs a multimodal feature of [$1024$, $4096$]. If the number of sampled frames is set to 128, then the modality features will be [$128$, $16$, $4096$], and the final multimodal feature will increase to [$2048$, $4096$]."}, {"review_id": "0Myjxsh7v1", "reviewer": "Reviewer_NaTe", "summary": "This paper presents TriSense, a novel triple-modality large language model with a dynamically adaptive Query-Based Connector that enables robust video temporal understanding across arbitrary modality combinations, supported by the TriSense-2M dataset, achieving state-of-the-art performance in segment captioning and moment retrieval under diverse modality dropout scenarios.", "questions": "Typo: In table 4, the second-best results are not underlined in ActivityNet-Caption.\n\n1.\tIn table 4, is the experiment only related to the number of frames? Can you give the results of other quantitative experiments such as inference speed with a generic scenario dataset?\n2.\tIn Section 3, TriSense-2M uses GPT-o1 for Judger training, but GPT-o1's closed-source nature limits reproducibility. Do you have tried to use open-source LLMs as an alternative? Can you give the effect of using different LLMs on the data?\n3.\tIn Section 3, have you calculated the percentage of missing modal data, and what specific modal training data affects the final model performance results?", "limitations": "The computational resources consumed by the model are too large, the experiment lacks interpretability, only the number of frames used is compared in the data comparison, and not the impact of more factors such as training data on the experiment, more adequate experiments and the same amount of data are needed for comparison to make the paper more convincing.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\nThe authors propose a groundbreaking adaptive weighting framework (Query-Based Connector) that dynamically reweights modalities under arbitrary missing inputs (vision/audio/speech), validated by TriSense-2M—the only dataset explicitly supporting modality dropout (2M samples), with best performance degradation under partial modality absence.\n\nWeaknesses\n1. The modal has computational efficiency concerns, the training demands are high: Stage 3 requires 16×A100 GPUs for 7 days, and also inference uses 64–128 frames per video, costing 8–10 hours for 11.4K test samples. While Slot-Based Compression reduces token counts, the LLM backbone and temporal modeling remain resource-intensive. No lightweight variants or quantization strategies are explored, hindering edge-device deployment.\n2. Potential bias exists in the dataset. TriSense-2M relies on GPT-generated captions for its Generator or Judger training. Manual filtering (500 samples/batch) is sparse, potentially retaining inconsistent samples and the paper lacks analysis of bias impacts.\n3. The modal in this paper underperforms in visual-only settings due to its multimodal optimization focus. This suggests the model prioritizes cross-modal synergy over single-modality refinement, limiting applications where audio/speech is unavailable. The authors acknowledge this but offer no mitigation strategy.", "was_revised": true, "final_justification": "I sincerely appreciate the authors’ patient response. My concerns have been resolved, and I would recommend incorporating the above experiments and analyses into the final version. I have increased my score.", "rebuttal": "**We thank all four reviewers for their constructive feedback:** \"The paper is clearly written and easy to follow, with well-organized sections and clear explanations of both the method and experiments\", \"The introduction of the new TriSense 2M dataset is a valuable contribution to the community\" (Reviewer ThZF); \"This paper studies a very interesting problem which seeks to build an omni-modal large language model for comprehensive multi-modal understanding\" (Reviewer 1DNL); \"The authors propose a groundbreaking adaptive weighting framework validated by TriSense-2M\" (Reviewer NaTe); \"Make substantial data contributions, which are significant for omni-model exploration\" (Reviewer 9tN8). **Next, we will carefully address each of the reviewers’ comments.**\n\n> Q1 & Q4: Training requires 16×A100 GPUs for 7 days; inference on 11.4K videos takes 8–10 hours (64–128 frames). Despite Slot-Based Compression, LLM and temporal modeling remain costly. No lightweight or quantized variants are offered. Table 4 lacks broader efficiency metrics.\n\nStage 3 involves full fine-tuning a 7B VLM on 2M samples, so using 16×A100 GPUs for 7 days **is standard** to training times reported in recent video-LLM works [1,2]. After CUDA/driver upgrades, training now reduces to **~5 days.** We plan to explore lightweight variants (e.g., quantization) moving forward. Table 4 is to demonstrate that TriSense can achieve strong performance on public benchmarks with **fewer frames**, so it is only related to frames.\n\nTo further clarify the computational efficiency, we conducted a comparison of inference performance against recent VLMs on a single H100 GPU using 64-frame input over 500 samples. Despite two additional modalities, our model achieves **competitive inference speed** and **outperforms** Qwen2.5-Omni with Flash Attention. Importantly, the inclusion of these modalities leads to significant omni-modal performance gains, thus a marginal trade-off in inference speed **is worthwhile**.\n\n| Model                         | Avg. CPU Time | Avg. GPU Time | Avg. Infer Time |\n| ----------------------------- | ------------- | ------------- | --------------- |\n| TimeChat                      | -             | -             | 1.33s           |\n| TRACE                         | 0.42s         | 1.02s         | 1.45s           |\n| Qwen2.5-Omni (w/o flash-attn) | 1.55s         | 6.21s         | 7.77s           |\n| Qwen2.5-Omni (w flash-attn)   | 1.58s         | 1.19s         | 2.78s           |\n| TriSense (Ours)               | 1.08s         | 1.16s         | 2.25s           |\n\n> Q2: TriSense-2M uses GPT-generated captions for training, risking bias. Sparse manual filtering (500 samples/batch) may let inconsistencies remain, with no analysis of bias impact provided.\n\nSince each event has four different caption versions and we set a **high acceptance threshold of 80%**, our actual workload was **4–5 times** higher than the basic sample count, so 500 samples per batch already resulted in a heavy workload for us. To overcome the potential bias, the pass rate for the judger was **set relatively low**. As mentioned at the end of **Section 3**, we started with **5 million** samples and ultimately selected only **2 million** after filtering. \n\nTo further validate annotation quality, we conducted a **human evaluation** on 100 randomly selected samples generated by the generator. Each was **scored** by a human annotator who **has not seen the model output** according to the rules in Figure 5. Lastly, **91%** samples are passed by human. The results show that while human captions were often more fluent, model-generated captions showed **strong** semantic adequacy and scalability advantages. Due to text constraints, we cannot provide more samples here, but we have provided **scoring samples** in the **response of Q3 for Reviewer 1DNL.**\n\n> Q3: The model performs poorly in visual-only settings because it is optimized for multimodal synergy rather than single-modality refinement. This limits its applicability when audio or speech is missing. Although this limitation is acknowledged, no mitigation strategy is provided.\n\nThe model achieves SOTA performance on **62.5%** of visual-only benchmarks, as reported in Section 5, indicating **strong competitiveness**. Therefore, the visual-only performance is **not degraded**, in the meantime, it is **also optimized for omni-modal flexibility**. To further strengthen visual-only capability, possible mitigation includes using a balanced ratio of omni-modal and visual-only samples (e.g., 90% and 10%), for which we are already taking actions. This may avoid overfitting to multimodal synergy while preserving single-modal ability.\n\n> Q5: Using closed-source GPT-o1 for Judger training limits reproducibility. It’s unclear if open-source LLMs were tested or how different LLMs affect the data.\n\nWe would like to clarify that, similar to previous works [1,2], closed-source models such as GPT have been **widely adopted**, as their instruction-following and captioning capabilities are generally recognized as superior to those of open-source models. To further improve reproducibility, we have included all our prompts in Figure 5 of **Appendix B**. Given the strong instruction-following ability of GPT-o1, and more importantly, the incorporation of human review, we believe that reproducibility **should not be a concern**.\n\nTo further evalute the generation quality of different models, we generated 100 omni-modal captions (AVS, VS, AV) with four different models and had human evaluators review each model’s outputs according to the standards in Figure 5. The results show that although open-source models like DeepSeek and Qwen3 can finish the task, their pass rates are **not as high as** closed models, and their outputs are **not sufficiently fluent or natural** in terms of English expressions. Therefore, it is necessary for us to **manually select high-quality data from closed-source models** and fine-tune the open-source models accordingly.\n\n| GPT-o1 (Close) | GPT-o4-Mini (Close) | DeepSeekR1-671B (Open) | Qwen3-235B (Open) |\n| -------------- | ------------------- | ---------------------- | ----------------- |\n| 92%            | 93%                 | 89%                    | 88%               |\n\n> Q6: In Section 3, have you calculated the percentage of missing modal data, and what specific modal training data affects the final model performance results?\n\nApproximately 10% of videos in TriSense-2M lack audio, in these cases, both audio and speech inputs are replaced with zero tensors. For the videos with audios, the model is forced to learn how to handle such dropout via dynamic modality reweighting, allowing robust performance under partial input conditions, as proven by our results in Section 5. \n\nTo assess individual modality contributions, we conducted an ablation study using 20K samples over 3 epochs training the model **from scratch**, selectively removing branches of the Query-Based Connector, benchmarking on randomly selected 500 samples. For example, **AV-Only** in the table below means we discarded the Speech branch. We initialize the LLM from [1] to avoid potential biases in TriSense’s LLM towards omni-modal connector. \n\nResults show that all three modalities **contribute positively** to all tasks, with the full AVS configuration performing best. Visual-only and dual-modal variants perform reasonably well, confirming graceful degradation. Due to resource and time constraints, we are unable to conduct larger-scale experiments at this stage, but we plan to include more in future versions of the paper.\n\n| Model    | AVS-SC                                       | VS-SC                                        | AV-SC                                        | V-SC                                         |\n| -------- | -------------------------------------------- | -------------------------------------------- | -------------------------------------------- | -------------------------------------------- |\n| V-Only   | **B:** 1.0 **M:** 1.7 **R:** 10.1 **C:** 0.9 | **B:** 1.1 **M:** 1.6 **R:** 10.3 **C:** 1.1 | **B:** 1.0 **M:** 1.7 **R:** 10.2 **C:** 1.1 | **B: 1.0 M: 1.8 R:** 13.3 **C:** 1.4         |\n| VS-Only  | **B:** 1.7 **M:** 2.9 **R:** 13.3 **C: 2.4** | **B:** 1.6 **M:** 2.9 **R:** 14.7 **C: 2.5** | **B:** 0.7 **M:** 2.3 **R:** 13.2 **C:** 1.5 | **B:** 0.7 **M:** 1.6 **R:** 13.2 **C:** 1.2 |\n| AV-Only  | **B:** 1.7 **M:** 3.1 **R:** 13.7 **C:** 1.9 | **B:** 1.4 **M:** 2.2 **R:** 12.0 **C:** 2.1 | **B:** 0.9 **M:** 3.1 **R:** 15.0 **C: 1.7** | **B:** 0.7 **M:** 1.6 **R:** 13.0 **C:** 1.1 |\n| AVS-Full | **B: 1.8 M: 3.4 R: 14.6 C: 2.4**             | **B: 1.7 M: 3.1 R: 15.1 C: 2.5**             | **B: 1.0 M: 3.2 R: 15.3 C: 1.7**             | **B:** 0.8 **M: 1.8 R: 13.4 C: 1.5**         |\n\n| Model    | AVS-MR                        | VS-MR                         | AV-MR                         | V-MR                          |\n| -------- | ----------------------------- | ----------------------------- | ----------------------------- | ----------------------------- |\n| V-Only   | **R@0.5:** 0.1 **R@0.7:** 0.0 | **R@0.5:** 0.1 **R@0.7:** 0.0 | **R@0.5:** 0.1 **R@0.7:** 0.1 | **R@0.5: 0.4 R@0.7: 0.2**     |\n| VS-Only  | **R@0.5:** 0.4 **R@0.7:** 0.1 | **R@0.5: 0.3 R@0.7: 0.2**     | **R@0.5:** 0.1 **R@0.7:** 0.0 | **R@0.5:** 0.3 **R@0.7:** 0.1 |\n| AV-Only  | **R@0.5:** 0.4 **R@0.7:** 0.2 | **R@0.5:** 0.1 **R@0.7:** 0.1 | **R@0.5: 0.3** **R@0.7:** 0.1 | **R@0.5:** 0.3 **R@0.7:** 0.1 |\n| AVS-Full | **R@0.5: 0.4 R@0.7: 0.3**     | **R@0.5: 0.3 R@0.7: 0.2**     | **R@0.5: 0.3 R@0.7: 0.2**     | **R@0.5: 0.4 R@0.7: 0.2**     |\n\n**References:**\n\n[1] Guo, Yongxin, et al. \"TRACE: Temporal Grounding Video LLM via Causal Event Modeling\". ICLR, 2025.\n\n[2] Guo, Yongxin, et al. \"VTG-LLM: Integrating timestamp knowledge into video llms for enhanced video temporal grounding\". AAAI, 2025."}, {"review_id": "D73bK472PC", "reviewer": "Reviewer_9tN8", "summary": "The authors design a video understanding model integrating audio, speech and visual modalities, build a three-modal caption dataset for training, and achieved promising results in temporal video understanding tasks, e.g., Moment Retrieval.", "questions": "* Why do the authors only focus on temporal video understanding tasks rather than more general video understanding benchmarks (e.g., VideoMME)?\n* The design of query-based connector to accommodate multi-modal inputs has been explored in prior work (e.g., Video-LLaMA). Is it appropriate to claim this as a core technical contribution at the model architecture level?\n* The authors aim to enhance video understanding (especially temporal tasks like Moment Retrieval and Segment Caption) by integrating audio-visual information, but lack ablation studies to demonstrate whether Speech/Audio modalities truly improve task performance.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strengths\n* Achieve promising results on temporal video understanding benchmarks.\n* Make substantial data contributions, which are significant for omni-model exploration.\n\n### Weaknesses\n\n* The authors focus only on simple and basic benchmarks/tasks, without considering implications for general video or audio-visual understanding.\n* The authors claim existing models fail to fuse audio information but doesn’t explain why. The proposed architecture resembles prior designs, suggesting the key contribution lies in the dataset.\n* The authors introduce audio/speech modalities without ablation studies or case demonstrations to validate their impact on temporal video understanding.", "was_revised": true, "final_justification": "The authors have addressed my questions comprehensively and clearly, and I have basically no further doubts about this paper. However, the dataset proposed by the authors does not consider verifying whether it can bring about the improvement of General Video Understanding capabilities, which may limit the extensibility of this work.", "rebuttal": "**First of all, we would like to sincerely thank the four reviewers for their constructive feedback on our manuscript:** \"The paper is clearly written and easy to follow, with well-organized sections and clear explanations of both the method and experiments\", \"The introduction of the new TriSense 2M dataset is a valuable contribution to the community\" (Reviewer ThZF); \"This paper studies a very interesting problem which seeks to build an omni-modal large language model for comprehensive multi-modal understanding\" (Reviewer 1DNL); \"The authors propose a groundbreaking adaptive weighting framework validated by TriSense-2M\" (Reviewer NaTe); \"Make substantial data contributions, which are significant for omni-model exploration\" (Reviewer 9tN8). **Next, we will carefully address each of the reviewers’ comments.**\n\n> Q1: The authors focus only on simple and basic benchmarks/tasks, without considering implications for general video or audio-visual understanding.\n\nWe would like to clarify that our task is **NOT simple**, we considered not only **audio-visual-speech tasks** and all their combinations (including audio-visual understanding), but also **general understanding** tasks (Table 10).\n\nFirstly, our proposed task requires the model to **natively support full-modality input** and perform temporal omni-modal captioning and moment retrieval, while also supporting **modality dropout**—features that most VLMs do not support. Secondly, as shown in Figure 3, we provide a detailed breakdown of video durations in TriSense-2M, where **83.5% of the videos are long-form (10–20 minutes)**, demanding accurate temporal understanding in long-form challenging settings. Finally, as shown by our experiments in Section 5, even models like Qwen2.5-Omni, which benefit from higher-quality training data and more training resources, struggle to perform well on our proposed task. This further demonstrates that our task presents a meaningful challenge for existing VLMs.\n\n> Q4: Why do the authors only focus on temporal video understanding tasks rather than more general video understanding benchmarks?\n\nWe would like to kindly emphasize that the dataset and model proposed in this paper are specifically designed to address the poor performance of current general VLMs in **omni-modal temporal understanding** scenarios. It is the temporal understanding that we aim to address, rather than general video understanding tasks. This is why we chose to focus on this direction. \n\nAdditionally, we have shown that the TriSense model not only outperforms existing VLMs in temporal understanding, but also **remains competitive in general understanding tasks**, even when trained with only **55%** of the general data, as we have included the **VideoMME** results in the Appendix C for your reference. We plan to include more general data in the future for a stonger model with both solid temporal & general understaning abilities.\n\n> Q2 & Q5: The authors argue that existing models struggle with audio fusion but do not explain why. Since similar query-based connector designs have appeared in prior work, is it appropriate to consider this a core technical contribution, or is the main novelty in the dataset?\n\nAnalysis was added in **Sections 1 (challenges) and 2 (last paragraph)** of the original paper, and more will be added in the supplementary material. Besides, we would like to respectfully claim that the proposed Query Based Connector **has NOT** been used by Video-LLaMA and VideoLLaMA 2, **especially for the modality reweighting mechanism,** with detailed explanations below:\n\n- **Architectural differences:**\n  1. **Video-LLaMA [1]** uses QFormer [2], which levergaes **fixed, learnable query vectors** to perform cross-attention across modalities, generating instruction-aware, fixed-length representations. In contrast, our method uses the **actual user-input prompt** as the query, making it flexible and only used in the **initial stage.** Because the cross-attention facilitates cross-modal interaction, placing the query alongside the target modality is a **natural and intuitive design choice.** Therefore, despite similarities, our approach **differs significantly** from QFormer in both the nature of the query and our overall objectives, as cross-attention is not the central contribution of our work.\n  2. **VideoLLaMA 2 [3]:** Although VideoLLaMA 2 also incoporates audio stream, the audio connector is just a simple MLP to project the audio feature, which is totally different from the Query-Based Connector proposed in our paper.\n  3. **Query-Based Connector:** The query attends to each modality (visual, audio, speech) separately via cross-attention to obtain query-relevant features (note: although this is similar to QFormer, it’s only our initial step). Then, all modalities pass through an MLP and softmax to obtain **dynamic weights** for each modality according to the query, enabling **modality reweighting** and temporal-aligned fusion that naturally masks out missing modalities. **Note that both of the Video-LLaMA or VideoLLaMA 2 have no modality reweighting mechanism.**\n\n- **Design Intentions Difference:** The primary purpose of QFormer is to extract single-modality features in order to **bridge the modality gap**. The self-gated Multimodal Query Fusion proposed by CREMA [4] is designed to **prevent the query token size from growing linearly with each new modality.** In contrast, our Query-Based Connector is designed to provide a **universal, simple, and effective way to extract features from all modalities and dynamically weight and fuse them**. This design philosophy naturally favors a simple connector structure, that is, making it too complex would risk reduced generalizability, overfitting, and optimization difficulties.\n\nLastly, we want to emphasize that our key contribution lies in demonstrating that such a concise and adaptable architecture can perform well across diverse modalities, which **has not been shown in prior work.** We believe this offers meaningful insights for future research. The proposed TriSense-2M dataset is intended to serve as a **robust pretraining foundation** for large multimodal models. Additionally, the TriSense model trained on this dataset can act as a **strong foundation model** for future studies, supporting fine-tuning and enhancement while substantially reducing training costs. Therefore, while our dataset is a key contribution, the Query-Based Connector is also a **core technical contribution**.\n\n> Q3 & Q6: The authors introduce audio and speech modalities to enhance temporal video understanding, but provide no ablation studies or case examples to show whether these modalities actually improve performance.\n\nWe provide the following ablation study **trained from scratch**. We selected 20K samples and trained for 3 epochs, removing certain branches of the Query-Based Connector. For example, **AV-Only** in the table means we discarded the Speech branch. During trainning, all the weigths are initialized from [4], and LLM backbone is set to frozen. Results show that all three modalities **contribute positively** to all tasks, with the full AVS configuration performing best. Due to resource and time constraints, we are unable to conduct larger-scale experiments at this time, but we plan to include them in future versions of the paper.\n\n| Model    | AVS-SC                                       | VS-SC                                        | AV-SC                                        | V-SC                                         |\n| -------- | -------------------------------------------- | -------------------------------------------- | -------------------------------------------- | -------------------------------------------- |\n| V-Only   | **B:** 1.0 **M:** 1.7 **R:** 10.1 **C:** 0.9 | **B:** 1.1 **M:** 1.6 **R:** 10.3 **C:** 1.1 | **B:** 1.0 **M:** 1.7 **R:** 10.2 **C:** 1.1 | **B: 1.0 M: 1.8 R:** 13.3 **C:** 1.4         |\n| VS-Only  | **B:** 1.7 **M:** 2.9 **R:** 13.3 **C: 2.4** | **B:** 1.6 **M:** 2.9 **R:** 14.7 **C: 2.5** | **B:** 0.7 **M:** 2.3 **R:** 13.2 **C:** 1.5 | **B:** 0.7 **M:** 1.6 **R:** 13.2 **C:** 1.2 |\n| AV-Only  | **B:** 1.7 **M:** 3.1 **R:** 13.7 **C:** 1.9 | **B:** 1.4 **M:** 2.2 **R:** 12.0 **C:** 2.1 | **B:** 0.9 **M:** 3.1 **R:** 15.0 **C: 1.7** | **B:** 0.7 **M:** 1.6 **R:** 13.0 **C:** 1.1 |\n| AVS-Full | **B: 1.8 M: 3.4 R: 14.6 C: 2.4**             | **B: 1.7 M: 3.1 R: 15.1 C: 2.5**             | **B: 1.0 M: 3.2 R: 15.3 C: 1.7**             | **B:** 0.8 **M: 1.8 R: 13.4 C: 1.5**         |\n\n| Model    | AVS-MR                        | VS-MR                         | AV-MR                         | V-MR                          |\n| -------- | ----------------------------- | ----------------------------- | ----------------------------- | ----------------------------- |\n| V-Only   | **R@0.5:** 0.1 **R@0.7:** 0.0 | **R@0.5:** 0.1 **R@0.7:** 0.0 | **R@0.5:** 0.1 **R@0.7:** 0.1 | **R@0.5: 0.4 R@0.7: 0.2**     |\n| VS-Only  | **R@0.5:** 0.4 **R@0.7:** 0.1 | **R@0.5: 0.3 R@0.7: 0.2**     | **R@0.5:** 0.1 **R@0.7:** 0.0 | **R@0.5:** 0.3 **R@0.7:** 0.1 |\n| AV-Only  | **R@0.5:** 0.4 **R@0.7:** 0.2 | **R@0.5:** 0.1 **R@0.7:** 0.1 | **R@0.5: 0.3** **R@0.7:** 0.1 | **R@0.5:** 0.3 **R@0.7:** 0.1 |\n| AVS-Full | **R@0.5: 0.4 R@0.7: 0.3**     | **R@0.5: 0.3 R@0.7: 0.2**     | **R@0.5: 0.3 R@0.7: 0.2**     | **R@0.5: 0.4 R@0.7: 0.2**     |\n\n**References:**\n\n[1] Zhang, Hang, et al. \"Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding.\" EMNLP, 2023.\n\n[2] Li, Junnan, et al. \"Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.\" ICLR, 2023.\n\n[3] Cheng, Zesen, et al. \"VideoLLaMA 2: Advancing spatial-temporal modeling and audio understanding in video-llms.\" arXiv, 2024.\n\n[4] Guo, Yongxin, et al. \"TRACE: Temporal Grounding Video LLM via Causal Event Modeling.\" ICLR, 2025."}]

Author rebuttal:

Meta-review: This paper makes two broad contributions: (1) a large multimodal model TriSense for video understanding that leverages audio, visual and speech modalities, (2) a dataset TriSense-2M built from existing benchmarks based on YouTube videos. In terms of the core technical contribution, the paper proposes a query-based connector that weights and thus fuses modalities based on their importance and their availability. Such "inequalities" in video data are also relfected in the proposed dataset. TriSense is evaluated on TriSense-2M as well as several standard public benchmarks to show its effectiveness.

All the reviewers and the AC appreciated the strengths of the paper, which include an approach to leverage multiple modalities for video understanding tasks and also an interesting dataset that is relevant for the research community. However, several questions on the novelty, relation to other fusion-based approaches, the quality of the proposed dataset, computational efficiency, potential biases, were noted in the initial reviews. The rebuttal and the discussion phases clarified all these concerns, resulting in the reviewers's overall positive evaluation of this paper. Thus, the AC recommends the paper for acceptance. The clarifications provided in the rebuttal and during the discussion phase should be included in the final version of the paper. The final version should also carefully address the ethical questions.
Decision: Accept (poster)