Can Vision-Language Models Answer Face to Face Questions in the Real-World?
Authors: Reza Pourreza
Rishit Dagli
Apratim Bhattacharyya
Sunny Panchal
Guillaume Berger
Roland Memisevic
Venue: ICLR 2026 Poster (2026)
Keywords: Situated Dataset
Multi-Modal Dataset
Vision Language Models
Primary area: datasets and benchmarks

Abstract: AI models have made significant strides in recent years in their ability to describe and answer questions about real-world images. They have also made progress in the ability to converse with users in real-time using audio input. This raises the question: have we reached the point where AI models, connected to a camera and microphone, can converse with users in real-time about scenes and events that are unfolding live in front of the camera? This has been a long-standing goal in AI and is a prerequisite for real-world AI assistants and humanoid robots to interact with humans in everyday situations. In this work, we introduce a new dataset and benchmark, the Interactive Video Dataset (IVD), which allows us to assess the extent to which existing models can support these abilities, and to what degree these capabilities can be instilled through fine-tuning. The dataset is based on a simple question-answering setup, where users ask questions that the system has to answer, in real-time, based on the camera and audio input. We show that existing models fall far behind human performance on this task, and we identify the main sources for the performance gap. However, we also show that for many of the required perceptual skills, fine-tuning on this form of data can significantly reduce this gap.

Published as a conference paper at ICLR 2026
CAN
MODELS ANSWER FACE TO FACE QUESTIONS IN THE REAL-WORLDY
Reza Pourreza'
Rishit D Apratim Bhattacharyya'
Sunny Panchal
Guillaume Berger'
Roland Memisevic'
Qualcommm AI Research* University of Toronto
"Equal contribution.
Work completed during an internship Qualcomm Al Research.
https:/ wpt.lccc.....ooooc.oooo////tttee
scte:acttseevv..ooo dataset-qird
Am using this right?
begit crssee
Was the first clup louder?
beg EnERee
right way pon
clap londer
many throw catch caught swies.
Itgie vnssant
Q: Is this my eye my nose? pointing
(begls onsiee!
What time does the clock say? 10:05 according clock.
many do my tosgue?? clicked your longue times.
Figure 1: We present the Qualcomm Interactive Video Dataset (QIVD), dataset collected in online guestion-answering setup, where users pose open-ended questions using their camera and microphone. QIVD offers videos with raw audio, annotated textual transcriptions of the spoken questions, and text answers with annotated timestamps. These timestamps indicate when a question can be sensibly answered given the video context. QIVD serves realistic and challenging dataset for situated visual reasoning in Large Multi-modal Models.
ABSTRACT
AI models have made significant strides in recent years in their ability to describe and answer questions about real-world images. They have also made progress in the ability to converse with users in real-time using audio input. This raises the question: have we reached the point where Al models, connected to camera and microphone, can converse with users in real-time about scenes and events that are unfolding live front the camera? This has been a long-standing goal in Al and is prerequisite for real-world AI assistants and humanoid robots to interact with humans everyday situations. this work, we introduce a new dataset and benchmark, the Qualcomm Interactive Video Dataset (QIVD), which allows us assess the extent to which existing models can support these abilities, and to what be instilled through fine-tuning. The dataset is based
degree these capabilities on a simple question-ansee answering setup, where users ask questions that the system has answer, in real-time, based on the camera and audio input. We show that existing models fall far behind human performance on this task, and we identify the main sources for the performance gap. However, we also show that for many of the required perceptual skills, fine-tuning on this form of data can significantly reduce this gap.
Qualcomm AI Research initiative of Qualcomm Technologies, Inc.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Recent advancements in Large Multimodal Models (LMM) have significantly enhanced the ability of Al systems to interact naturally and fluently with users in real-time, Existing AI agents can process audio, speech, and visual inputs to engage in conversations about images or videos. However, the conversational capabilities of kate-of-the-ar LMMs such as GPT-4o (Hurst et al. (024) are limited to queestion-answerin
on visual understanding and reasoning tasks, such as describing images or answering questions that require inferring object positions and relations in the visual input. These systems often fail to provide truly situated, live, conversational experiences (Figure [ that we may expect from humanoid robots real-time video-call chatbots in the future.
We hypothesize that this limitation stems from the fact that current vision-language datasets and benchmarks are biased toward offline reasoning about images and videos. That is, the models receive the entire visual input and the entire question at once before being required to provide an answer. This is because the training data for such tasks can be easily sourced on the internet or casily generated through automated pipelines. There is a distinct lack of benchmarks and datasets that test genuine, real-time, "face-to-face" conversational skills. A separate but related problem is that models not trained to respond at the appropriate time in a conversation - knowing "when to speak" is crucial for conducting real-world conversations, yet this timing skill remains underdeveloped and understudied in current benchmarks.
To address these challenges and assess the limitations of existing models, we introduce the Qualcomm Interactive Video Dataset (QIVD), new dataset and benchmark designed for end-to-end trained systems aimed real-time user interaction. QIVD is structured as an online question-answene where users pose open-ended questions using their camera and microphone, and the system must respond appropriately. Our work differs fundamentally from other related datasets and benchmarks by introducing an entirely online guestion-answering paradigm where both questions and answers evolve in real-time as the video unfolds, requiring models to maintain contextual awareness while handling inherent ambiguities in human references visual elements. We show how this simple type of interaction allows us to capture a rich set of visual concepts that fall under the umbrella of situated visual understanding, including deictic (referring) expressions, pointing gestures, object ambiguities, behavior and action understanding, and counting, as well as audio-visual concepts. An overview of dataset is shown in Figure [1 Due to the in-the-wild nature of the recordings, the videos exhibit considerable variation in lighting conditions, background settings, the range and nature of questions posed, actions performed by subjects, and other audio-visual characteristics.
setup,
To showcase the unique challenges our dataset presents, we conduct series of experiments where we evaluate multiple open and closed-source state-of-the-art models, and fine-tuned models on our dataset. Our experiments reveal that the seemingly simple interaction of answering questions live, in real-time, is highly challenging for existing Al systems (Hurst et al., 2024), even if they are otherwise good at performing visual reasoning. Our experiments indicate that the failure modes of existing systems can be attributed to their: (1) difficulty integrating visual and auditory information in real-time to disambiguate questions, (2) inability to determine the appropriate time at which to answer, and (3) inability to answer questions whose answers require situational common sense. Our dataset supports research on online LMMs capable of situated audio-visual reasoning and can be leveraged to build conversational agents that interact with users in real-time.
Our contributions are summarized follows:
1. We introduce QIVD, a novel multi-modal dataset designed to evaluate online situated audio-visual reasoning and real-time conversational skills
2. We benchmark existing LMMs and identify critical weaknesses in their ability to handle real-life conversations.
3. We demonstrate that these limitations can be effectively mitigated by fine-tuning models on appropriate audio-visual conversational data.
4. We develop a simple yet effective baseline to process streaming audio-visual inputs, depart- ing from traditional offline paradigms.
qual comm. ccmm.ccm//ssveelgerr///fttoorrrr - vivd
comm-
2

---

Published as a conference paper at ICLR 2026
Table Comparison of various benchmarks encompassing several key aspects.
Benchmark
#Videos
#QA
Anmotation
Interactive
AVSD (DSTC7) Alamri et al
2018
11156
~111560
Manual
Knowll YQA
Gareaa 2 2020
207
24282
Manual
x
LifeQA Castro ai. 2020
275
2326
Manual
How2QA ar 2020
9035
44007
Manual
MedVidQ Luppa @ W
2025
899
3010
Manual
J
Social-1Q
Zipgin
2019
1250
7500
Manual
X
Video-MME Fuu
20124
900
2700
Manual
X
CodeVidQA Kaja si.
2025
2104
2104
Automatic
Ego4D Social Inferrcttions
al.
2022
667
task-spocific fabels
Manual
J
TVQA (Lci 2018
21793
152545
Manual
NEXT-GQA Aiio et 2024
1557
10531
Manual
STAR al. 2024
22000
60000
Automatic
VStreaMI-QA 22hing
2024
32
3500
Automatic
QIVD
2900
2900
Manual
2 RELATED WORK
Offline Video Evaluation Benchmarks: Prior work video understanding benchmarks has primar-
ily focused on offline evaluation paradigms. There have been multiple temporal video understanding
benchmarks for open-domain understanding (Li et al. [2024b] Liu et al. 2024bb Xu et al., 2023
Patraucean et al.) 2023 Ning et al.] 2023 Hu et al. (025). hand movements (Goyal et al. 2017
Materzynska et al. 2019), articulated motion (Dagli al. (024), full human body motion (1 (Panchal
et al. 2024). robotics (Haresh et al. 2024 Yu et al.) 2024] Bao et al. [023a Gu et al. 2023 Brohan
al.
2023 Jiang et al. 2022), and embodied reasoning (Yang al. (025b). These benchmarks
wate models' ability to comprehend temporal relationships but operate in a fully offline manner.
Long-form video understanding has been addressed by datasets such as LVBench (Wang et al., 2025).
and MoVQA (Zhang et al.) 2023b), which extend the context window but fail to simulate real- time constraints. contrast, our QIVD dataset and benchmark focuses on real-world QA.
Situated Video Evaluation Benchmarks: Situated question answering has also been studied by (Das
cet al. 2018] Ma al., 2023) and follow-up works (e.g. (Wang al.) [2023] [2026] Wu al. 2023)). separate line of work has studied "common sense" situational understanding for Al models, albeit not VQA format. This includes the work by (Goyal et al. 2017 et. al., 2022] Patraucean et al., 2023) and recent work situated live dialogue (Bao al. 2023b_lo Panchal 2024). Our work is similar in that involves real-world interaction. In contrast to the existing work, questions in our dataset are free- free-form and open-ended rather than task-specific and oriented toward a specific goal.
In contrast to existing question- aattiionnnnwerrig tasks, the task introduced in our work involves real-world interaction with user, and as such the input is not confined to only visual information. Moreover, we place the task into a truly situated context, where correct answers require true understanding of the scene unfolding in the real world. In contrast that line of work, in this paper, we study situated question answering in a real-world not synthetic environment, by interacting "live" with a human subject, and by using audio and video input.
Online Models: Recent work on online video processing includes LidooLLLe (Chen et al.
2024) and FStrhn (Zhang al. 2024), which attempt address real-time processing con- straints but remain limited in their ability to handle deictic references and situated understanding and also do not include audio. The Stream VLM (Panchal et al. 2024) supports situated understanding but is limited to the fitness domain and also lacks audio. Furthermore, existing benchmarks typi- cally evaluate general visual understanding rather than modeling the situated, interactive nature of real-world human-Al conversations about visual content.
QIVD
The purpose of QIVD is to train and evaluate Al models on situated visual understanding. Each data instance comprises video sequence annotated with temporally synchronized -anste pairs. Additionally, the dataset includes the ground-trun -ruth answer to the question, making it possible to probe model's understanding of the situation depieted in given clip. Structuring the data as simple
3

---

Published as a conference paper at ICLR 2026
question-answering task allows us separate situated understanding from multi-hop conversational capabilities. The latter is a similarly difficult but largely orthogonal challenge for existing models. A side-by-side comparison of the features offered by QIVD and other related datasets is presented in Table
3.1 DATA COLLECTION
Recording: Crowd workers were instructed to record short videos using the camera and microphone of their mobile phone or laptop. They were free to choose the content of their videos but were shown examples featuring various gestures, actions, and objects to help them understand the dataset's s purpose. The participants received written instructions explaining that these videos would be used to train and evaluate Al systems in understanding visual scenes. The instructions clarified that the AI system's 's purpose would be correctly answer single question rather than engage in multi-step conversation. While recording their vidcos, crowd workers posed a question related to what was being shown. They were encouraged to be creative with their questions while ensuring they referenced the action or scene being recorded. After collection, all videos were inspected for audio and video quality, and their suitability for inclusion in the dataset.
Each video in the QIVD dataset has three annotations, First, it includes a human- coman-generated transcript of the question asked during the recording. Second, we provide a numan-generate answer to that question. Third, we added a timestamp that marks the specific moment when would be appropriate to answer the question. The timestamp does not always coincide with the end of the spoken question-in many cases, additional video context is required after the question was asked. For example, if a participant asked "What action is this?" before performing the action, the appropriate moment to answer would be after the action was visible in the video. This approach ensures that annotations also reflect when sufficient information becomes available to answer the question correctly, if required, rather than simply when the question ends. Finally, all of our submissions were reviewed by humans to verify their accuracy.
Annotation Methodology:
Unlike datasets constructed from pre-recorded videos with post-hoc annotations, contempo- raneous question-asking approach places a strong demand on situational context understanding, Our videos capture authentic uncertainty about future events in the video, including questions that genuinely test temporal reasoning, and require situational awareness to answer at the appropriate time. The annotations for answer timing are particularly valuable as they acknowledge that certain queries require monitoring the audio or visual stream over time to aggregate relevant information, and ascertaining when to respond. Through our collection approach, we provide a robust benchmark for evaluating model's proficiency in understanding and responding to situated We show few examples from our dataset Figure I Jusin four frames per video.
stimuli.
3.2 POST-PROCESSIA WORKFLOW
Following the initial data collection phase, we perform comprehensive post- post-processing enhance dataset utility by adding structured metadata and further ensure dataset quality. This section details our approach to quality assurance and taxonomic categorization of the dataset.
To ensure data quality and ethical standards, we used a multi-stage quality control process. Each video underwent automated evaluation followed by manual inspection by trained evaluators who assessed the content according to predefined exclusion criteria. Specifically, we examined all videos for the presence third persons, private data, and protected intellectual property: for the presence of inappropriate content, such as hate speech, and other potentially harmful elements; for linguistic compliance (clearly intelligible, English audio content); and for technical quality (absence of severe motion blur, compression artifacts, etc.).
Quality Checks:
After inspection, 2900 videos were deemed suitable and included in the dataset.
Semantic Categorization: To facilitate fine-grained analysis of model performance across different visual reasoning tasks, we developed a taxonomy of question types. The taxonomic structure allows for systematic evaluation of model performance across diverse visual reasoning tasks, enabling to identify specific strengths and weaknesses in situated understanding capabilities. Each qudot-uuetioon pair was assigned one more of 13 predefined semantie categories representing distinct visual
4

---

Published as a conference paper at ICLR 2026
Table 2: Dataset size metries (total videos, vocabulary size), video characteristics (total frames, average length, frame rate, resolution), and linguistic properties of questions and answers. Average answer timestamps are represented by the average time in the video when the question should optimally be answered percentage of the video duration. The token statistics are calculated with the Llama-3 tokenizer. Standard deviations are shown in parentheses.
Statistic
Statistic
Value
Answer Tanestamp
81.47% 13.89)
Vocabulary Size (words)
3624
Avg. FPS
30 (::0.00)
(lokens)
3U72
Question Types (Total)
Total Frames
443350
Questions with "where"
47
Avg. Video Length (s)
5.10 (:::0.4))
Questions with "bow"
512
Avg. Question Length (wards)
(:11.94)
(tokens)
7.60 (::2.228)
Questions with "what"
1102
Avg. Answer Longth (words)
7.23 (::4.31)
Deictic References (Tonal)
(lokens)
9.73 (:::.61)
Questions with
Avg. Short Answer Length (wards)
1.38 (:1.0.2)
Questions with "these"
39
(lokens)
1.98 (:2:1.27)
Questions with "that"
Avg. Resolution (width) (height)
640.00 (:1:0.00)
Questions with "there"
105
(1.46.01)
Questions with "this"
568
Table 3: Distribution of samples across the 13 semantic categories our dataset, with the answer timestamp a percentage of video duration for each category. Percentages in the Samples column show the relative distribution of categories within the dataset.
Category
Answer Timestamp
Samples
Category
Answer Timestamp
Samples
Attributes
(3.56))
(5.34%)
Object Referencing
79.18%
13.61)
(24.34%)
Action Counting
92.22% (+:8.73)
225 (7.76%
Object Understanding
80.63% (#: 14.07)
79 (2.72%)
Action Detection
85.46% (=13222)
(15.17%)
Scene Understanding
79.91%
13.58)
(1.31%)
Action Understanding
81,47% (=15.07)
(3.79%)
Audio-Visunl
90.09% (1:11.49)
(0.76%)
Object Aliributes
79.52% ( 13.41)
562 (19.38%)
OCR
83.04% (11 13.088
(0.79%)
Object Counting
78.41%
(= 12.75)
286 (9.86%
Subjective
77.39% (:: 15.155
43 (1.48%)
Object Detection
76.95% (= 15.65)
211 (7.28%
Total
81.47% (3.899
2900 (100%)
reasoning capabilities. The categorization process uses a semi-automated approach: first, large language model (LLM) is used to perform preliminary classification based on question content and transcribed answers; next, human annotators verify and refine the categories. Our semantic taxonomy encompasses the categories listed in Appendix B.2
Answer Normalization: To facilitate better quantitative evaluation and reduce ambiguity in model assessment, we implemented an answer normalization process. For each original free-form response, generated condensed "short-answer" version that retained only the essential information re- quired to correctly address the question. We follow a similar cemi-automated method as semantic categorization for generating short answers. During evaluation, use both the short answer and the original ground truth to evaluate models.
3.3 DATASET STATISTICS
The QIVD dataset consists of 2900 video clips and thus 2900 unique question- answer pairs. Table summarizes the statistics of the dataset. The majority of clips have length between 4 and 8 seconds. This range captures the natural timeframe in which situated question about the visual scene can be posed and answered. We show the breakdown by the semantic taxonomy (Section 3.2) of the question-answer -nnwer pairs in Table
Dataset Composition:
Temporal Characteristics: A distinctive feature of the QIVD dataset is the temporal relationship between the point in time when a question is posed and the point in time when sufficient information available to answer We analyze the temporal characteristics by category in Table which shows the distribution of optimal answer timestamps relative to video duration for each category. As we observe from Table 3, retate- categories generally require observing larger portion of the video before answering, with Action Counting showing the highest optimal time (92.22% of video duration). This reflects the natural temporal dependency in action-related questions, where the answer often depends observing the completion of an action sequence. In contrast, Object Detection
5

---

Published as a conference paper at ICLR 2026
Table 4: ASR performance comparison. Evaluation of Automatic Speech Recognition (ASR)
systems on the QIVD dataset using standard text similarity metrics. Timestamp detection accuracy is measured by At that represents the mean absolute error in the optimal time answer.
Model
METEOR BLEU
ROUGE-LT
Art
4 A 4++4
Whisper [Radford al
2023
90.01
80.95
90.32
Whisper-Streaming Machlook etal.
2023
92.34
74.57
91.82
0.83
-0.9
0.61
Stream- Fream-Qweo-Omnl
0.52
-0.62
0.53
(76.95% of video duration) and Subjective questions (77.39% of video duration) can typically be answered earlier in the video, often right after the question is asked.
BASELINE STREAMING APPROACH
Two critical features of QIVD are:
The videos are self-contained, with the question embedded in the audio channel. An optimal model should be capable of answering these questions directly from the videos without the need transcription.
Self-Contained Videos:
When-to-Answer Desiderata: The videos are sufficiently long to include a scenario, a question, and any additional frames. An effective streaming model should identify the ideal moment to start answering the question, which is when both the question and any information necessary to answer it are present.
Current offteoofthhe-art LMMs do not integrate streaming and concurrent processing of audio and video information for situational interaction. To address this gap, we propose novel streaming approach that pairs streaming automatic speech recognition (ASR) system, used to transeribe questions and detect answer moments, with a Video-LMM to analyze video content and provide answers.
In detail, our streaming approach relies on the Streaming- Freammnn--Whispe model (Machácek et al. 2023) to identify "when answer". The Streaming- Whishpr model (Machácek al. 2023) LocalAgreement algorithm (Liu et al., (2020) to transcribe text in a streaming setup. The LocalAgree- ment algorithm transcribes the input audio in chunks and subset of previous chunks are used to condition the transcription of the next chunk. In practice, we found that chunk size of 0.25 seconds sufficient for accurate streaming transcription for this dataset. Processing the input audio in chunks allows us to detect the end of the question asked by the participant in the video. It is important to note that, as mentioned above, the end of question does not necessarily capture the optimal moment for an answer, as some necessary information may arise later in the video. Thus, we consider this approach as a reasonable compromise given the current limitations of ASR solutions and LMMs. After detecting the end of the question, the input video and audio up that timestamp, along with the transcribed question, are provided as input to the LMM backbone. The LMM backbone can then process the multi-n modal video and audio inputs along with the transcribed question provide answer. We explore different LMM backbones as outlined Section 5.1
the
EXPERIMENTS
We conduct comprehensive experiments to evaluate various open- and closed-source models on QIVD.
5.1
EXPERIMENTAL SETUP
Configurations:
The experiments performed within four distinct setups:
Streaming setup: Under this setup, we evaluate the baseline streaming approach introduced in Section

---

Published as a conference paper at ICLR 2026
2. Offline setup: In the baseline streaming approach, evaluating LMMs can be challenging due to potential inaccuracies in the questions extracted by the streaming ASR system, leading to accumulated errors. Therefore, in the offline setting, we use ground-truth questions to evaluate the models. This approach ensures that the evaluation is based on perfectly transcribed questions, allowing for an effective assessment of merely model's answering performance. Consequently, the performance is an optimistic estimate of overall real-world performance.
3. Impact of audio: Among existing LMMs, the VideoLLaMA family of models (Zhang et al. 2023a) are state-of-the-art models capable of simultaneously processing both audio and video content. Although these models cannot transcribe speech, they can utilize audio content complementary source of information, thereby potentially enhancing accuracy. We evaluate these models by examining the impact of additional audio on the accuracy of their question-answeriing capabilities.
4. Impact of when-to to-answer: This experiment investigates how the timing of the when to answer moment affects model performance. We utilize the Qwen2.5-Omni model (Xu al. 2025). the only publicly available model capable of concurrently processing both audio and video modalities while also transcribing speech. The model provided with both the ground-truth and ASR- derived when-to-answee timestamps. It then transcribes the question and generates answer, allowing us to compare the outputs and assess the influence of timing on response quality.
Baseline Models: We experiment with various open-source and closed-source LMMs.
The open-source models we evaluate include InstructBLIP (7B) (Dai et al., 2023), Video-ChatGPT
(7B) (Maz et al. 2024), VideoChat (7B) > (Li et al., 2024a), VideoChat2 (7B) (Li et al. 2024b), LLa
NeXT (7B) (Liu et al. 2024a), LLaMA- VID (13B) (Li et al. 2024c), Video-LLaMe
(13B) (Zhang
ct al. 2023a), VideoLLaMA2 (7B/72B) [Cheng et al. 2024). VideoLLaMA2.1 (7B) (Cheng et al.
2024).
VideoLLaMA3 (7B) (Zhang et al. (2025), Video-LLaVa (7B) (Zhu et al. 2 Lin et al.
2023). VideoL.LM-Online (Chen al. (024)) Flash-VStream VStream (Zhang et al. 2024). Chat-Unity
(13B) eJ 2024), Qwen2.5-VL. VL (7B) (Wang et al. (2024). Qwen2.5-Omm (7B) (Xu et al. 2025).
Qwen3-VL (8B) (Yang al.] 2025a) The model sizes range from 7B 13B parameters for the
language backbone, with the exception of VideoLLaMA2-722 (Cheng et al. 2024). All models are evaluated in a zero-shot setting. We utilize the vision and audio heads provided with the checkpoints to process the input. For InstructBLIP (Dai et al. 2023). an image model, we sample 4 frames from each video, process these frames with the image encoder and a Q-Former (Zhang et al., (023c) as individual images, and then treat all features as a long sequence of image tokens for the language model.
Additionally, we evaluate closed-source model, GPT-4o (Hurst et al. 2024) and Gemini-2.5. Flash (Comanici et al. 2025). in zero-shot fashion. For GPT-4o, videos are preprocessed by uniformly selecting 4 frames from each video and down-scaling the resolution to half. The query used to prompt GPT-4o is provided in the appendix.
Since the answers in QIVD are in free-form, we determine the correctness of an answer using an LLM judge that receives a question, the ground-truth answer, and the predicted answer, alongside the short answer and the category of the question, and determines if the predicted answer is correct. We use a pre-trained Qwen3-8B model (Yang et al.) (025a) as the LLM judge (see Appendix C.6 for comparisons with other LLM judges). The prompts that were used are provided in the appendix. In addition, we report Bert (Zhang* et al. 2020). METEOR (Lavie & Agarwal 2007). BLEU (Papineni al.) (2002), and ROUGE (Lin 2004) scores between the ground-truth answers and the predicted answers.
Evaluation Metries:
5.2 RESULTS
We now present the results obtained from the three settings described in Section 5.1
Streaming setup: Table 4 presents the transcription results obtained from Whisper-
Streaming (Machácek et al. 2023). where the transcription quality is quantified using BLEU (Papineni
et al. 2002), ROUGE (Lin 2004). and METEOR (Lavie & Agarwal 2007) scores, by comparing the transcribed questions to the ground-truth questions. The when- thenntt-wee answer metric. denoted by At in the table, is measured as the Mean Absolute Error between the time-to-an to-wweer extracted by Whisper-Streaming and the ground-truth value. In addition the overall MAF, we also report the
7

---

Published as a conference paper at ICLR 2026
Table 5: Evaluation of baseline LMMs on the QIVD dataset using (a) questions and estimated when-to-answer timestamps by Whisper (Radford et al. (023) and (b) ground-truth questions and timestamps. Corr. represents correctmess by the LLM judge
ASR Questlons und Timestamps MUTBOR
Human Questioas and Timestamps BLEU
Model
Con:
BERT
DLDU
ROUGE-L
Corr.
BERT
METDOK
ROUGELL
Chat UtiVi ml RXA
34,06
89.91
37.47
28.45
00.79
90.50
40.02
7.24
31.22
InstructDL. [2
35.03
82:19
4.35
39.14
$2.00
10.72
LLaMA-VID.
39.41
90.51
37.19
29.
43.0
90.78
37.55
5.42
29:
LLAVA-N6X1
85.29
22.85
55.78
24.50
Video ChaiGe
90.33
38.13
31.08
36,59
91.01
40.59
33.55
VikaChau
85.05
23.48
12.22
3.92
85.20
24.30
12.54
VidooClutz
11.35
47.93
12.43
Video-LLl.V
15.0
Yarapeeri Eiteker
89.50
VideoLLaMAL
VideoLLaMA2
VideolLaMAA-70
VideoLL
33.08
28.95
27.05
30.79
Qwen2.5-V1..
Wattr
37.37
Owen2.5-Quaii
33.45
33.98
Qwen1-VL-B8
53.72
87.08
31.53
80.07
87.58
36.73
35.89
Grmini-2.5.
58.07
43.07
GPT-40 Pland
58.76
89
51.18
Human (sobset)
93.01
53.21
17,40
Videol I L-MA2,1-7B-V
Videol.1 Madlol..aMA..II LMA2, 1-7B-AV
Videol. 1-72.1-7B-FT
Videol I 1--F.I-7B-FT
(ocallated on video)
(ecalastid) un sudlovidao)
Correctnoss
50
60
40
Action Atmibutt
Actionn
Action Detection
Orject Altribates
Objecct Couning
Olyeat Refemne- ing
OCR
Same Under standing
Subyarivv
Ovenil
Counting
Under-
Visral
Detection
Unsla- saading
standing
Evaluations of the public and finetuned VideoLLaMA2.1171B- vision + audio and vision-only settings.
Figure 2:
(Cheng et al. 2024 in
mean values of both negative and positive At instances. Notably, the average negative At is larger in magnitude, suggesting that models which initiate responses immediately after detecting the end of question tend to answer prematurely-ffir received. We also report the results obtained from the standard Whisper model (Radford et al.) 2023) additional baseline. is worth noting that this model does return any timestamps alongside the transcriptions.
before sufficient contextual information has been
The baseline LMMs evaluated using video trimmed the when-to-answer timestamp and the question, both extracted via Whisper-Streaming (Machácek et al., 2023). Table summarizes the baseline results.
For the evaluation in the offline setup, we provide the baseline LMMs with a video that is trimmed the ground-trutt when-to-answer timestamp alongside a ground-truth question. We summarize these results in Table Additionally, we engage a non-expert human annotator to re-annotate random subset of the dataset containing 300 samples, establishing human baseline.
Offline setup:
The only publicly available checkpoint from the VideoLLaMA Cheng et al. 2024 family that supports concurrent audio and video processing is VideoLLaMAZ.1-73--3 (Cheng et al. 2024). We evaluate this model using ground-truth transcribed questions in two distinct settings. In the first setting, we provide the model with both audio and visual information, while in the second setting, we supply only visual information. The results, depicted in Figure show setting (1) in red and setting (2) in blue Interestingly, and contrary to expectations, the model's performance degrades with the addition of audio information.
Impact of audio:
We fine-tune this model QIVD using both audio and video modalities. Due to the dataset's 's small size, we apply 5-fold cross- volssvvaiia.iin. The vision encoder remains frozen, while the LLM backbone and audio pathway are fine-ture fnnet for two epochs per fold. We repeat the initial experiments with
8

---

Published as a conference paper at ICLR 2026
Timestamps
Timedtaps
Stream
Timestarps
Accuracy 196)
22L
2.5
20
52
Action Artraaries
calntiing
Action Desectinn
Acten Und Understanniig
Dbject rivs
Object Coanting
Dblecs
oblr Peteerncing
Un Untesstansing
Odjes
ock
Undernanninng
Scere
Subbecthe
Overal
Figure 3: Evaluations of Qwen2.5-Omni (7B) with when-to-answer moment provided from different sources.
the fine-tuned model. Results in Figure show setting (1) in purple and setting (2) in green The fine-tuned model performs best when both modalities are available but underperforms the pretrained model in some video-only cases, likely due its adaptation to multimodal inputs. Since QIVD relies heavily on audio cues, missing audio during inference significantly impairs performance.
Impact of when-to-answer: We adapted Qwen2.5-Ommi (7B) (Xu et 2025 to support streaming format by feeding the model chunked audio data, with each chunk representing one second. The model was fine-tuned detect when-to-answe by emitting special token during listening and generating response once the appropriate moment was detected. Technical details are provided in Appendix D.3 The when-to-answer accuracy Stream-Qwen-Omee
reported in Table 4 showing To assess the impact of accurate timing, we evaluated Qwen2.5-0mm using timestamps from three sources: ground -tutt (GT) annotations, ASR-Stream predictions, and Srream--wen2.5-000 of answers generated by Stream-Qwee Seeinn--een--iee Ommi. As shown Figure 3 precise estimation of the when-to-answe moment substantially improves overall model performance.
significant improvement over
predictions. We also report the correctness
5.3 DISCUSSIONS
To facilitate more comprehensive analysis of the strengths and weaknesses of the baseline LMMs. we compare the correctness of selected baseline L.MS across individual categories of QIVD, as illustrated in Figure The human baseline is derived from a small subset of the data, detailed in Section 5.2 As demonstrated in Table 5 and Figure 4 there is a significant performance gap between non-expert human and all the models, including state-of-the-are systems, across all evaluation categories. Humans demonstrate near-perfect performance in categories where Al systems struggle significantly, particularly in action counting, audio visual integration, and object referencing. This disparity is most pronounced in tasks requiring temporal reasoning and deictic reference resolution, where humans outperform the best Al system by large margin.
Furthermore, as shown in Figure the baseline models exhibit inconsistent capabilities when faced with various types of situated visual reasoning. While these models perform reasonably well on basic object detection tasks, their performance declines markedly on tasks involving action counting, temporal sequencing, and audio-visue integration. This capability gap indicates that current models are optimized for static scene understanding rather than the dynamic temporal reasoning required for real-time interaction scenarios.
The most common failure modes include: (1) misinterpreting deictic references, (2) incorrect action counting, (3) temporal sequencing confusion, and (4) audio-visual misalignment. Many of these failures occur regardless of model size or architecture type, suggesting fundamental limitations in current approaches to multi-modal integration rather than just capacity constraints.
9

---

Published as a conference paper at ICLR 2026
Our fine-tuning experiments show that the performance improvements from fine-tuning not distributed uni- formly across task categories. As shown in Figure 2] fine-tuning pro- duces the most dramatic gains in action counting (+16.96%), action understanding (+10.00%). subjec- tive (+23,26%) and audio-visual (+17.39%) tasks, while yielding mini- mal improvements in object attributes (+1.24%) and scene understanding (+2.63%). This asymmetric benefit pattern suggests that certain situated understanding capabilities are more amenable to data-driven adaptation than others. Particularly, even after fine-tuning, performance on action counting remains very low (29.91%) 1)). indicating that these temporal reason- ing capabilities may require more so- phisticated architectural inductive bi- ases.
Figure 4: Correctness of selected baseline LMMs across QIVD categories (best viewed with zoom).
As shown in Figure the integration of audio and visual modalities results
substantial performance gains across nearly all task categories. The VideoLLaMA2.117B--
model shows significant improvement over its vision-only counterpart in visiol tasks would expect. However, this improvement extends beyond explicitly audio-related tasks, with notable gains in subjective (+37.61 1)), object detection (+9.48%). and object counting (+10.14%) These findings empirically confirm our hypothesis that existing vision-language systems are fundamentally limited by their modular pipelines that process visual and audio information separately. We show that end-to-end multimodal training creates emergent capabilities that transcend simple feature concatenation, enabling more sophisticated situated understanding in real-time interactions.
CONCLUSION
We introduce QIVD, a comprehensive benchmark and dataset designed to assess and train LMMs (video. audio, and language) on a wide variety of tasks that require responding to humans in real time. Through extensive experiments, we identify key challenges with existing models for situated visual understanding. Our dataset follows simple question-answent situated understanding capabilities without being confounded by the need for multi-hop conversational capabilities. The dataset also does not require any domain- -specffc knowledge or complex reasoning skills. Yet we show that the task is still highly challenging for LMMs. Based on these insights, we hope that QIVD will inspire and guide future research, driving the development of Al systems that can interact with humans in realistic scenarios in an online fashion.
paradigm and thereby tests for
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Huda Alamri, Vincent Cartillier, Raphael Gontijo Lopes, Abhishek Das, Jue Wang, Irfan Essa, Dhruv Batra, Devi Parikh, Anoop Cherian, Tim K Marks, et al. Audio visual scene-aware dialog (avsd) challenge at dstc7. arXiv preprint arXiv: 1806. 000000 2018.
Chen Bao, Helin Xu, Yuzhe Qin, and Xiaolong Wang. Dexart: Benchmarking generalizable dexterous manipulation with articulated objects. In CVPR, 2023a.
Yuwei Bao, Keunwoo Peter Yu, Yichi Zhang, Shane Storks, Itamar Bar- Yossef, Alexander De La Iglesia, Megan Su, Xiao Lin Zheng, and Joyce Chai. Can foundation models watch, talk and guide you step by step to make a cake?, 2023b.
Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, et al. Rt-2: Sasion-lnguaae-actiio models transfer web knowledge robotic control. arXiv preprint arXiv:2307. 15818, 2023.
Santiago Castro, Mahmoud Azab, Jonathan Stroud, Cristina Noujaim, Ruoyao Wang. Jia Deng, and Rada Mihalcea. Lifeqa: A real-life dataset for video question answering. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4352-4358, 2020.
Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia- Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In CVPR, 2024.
Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. Beats: Audio pre-training with acoustic tokenizers, 2022. URRphttts:///rxi..ooo org/aborggab. /abs/
09058
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo. Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv preprint arXiv:2406.07476, 2024. URL
hittss://ara
or.. abs 2406 07476
Gheorghe Comanici, Eric Bicber, Mike Schackermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang. Evan Rosen, Luke Marris, and Others. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025. Utt/tttss://axxi org/ abs abs 2507. 06261
Rishit Dagli, Guillaume Berger, Joanna Materzynska, Ingo Bax, and Roland Memisevic, Airletters: An open video dataset of characters drawn in the air. In ECCV Workshops, 2024.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. InstructBLIP: Towards general purppss languaeee models with instruction tuning. In NeurlPS, 2023.
Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, and Dhruv Batra. Embodied question answering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Grattafiori et. al. The llama 3 herd of models, 2024.
Grauman et. al. Ego4d: Around the world 3,000 hours of egocentric video. In CVPR, 2022.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, Peixian Chen, Yanwei Li, Shaohui Lin, Sirui Zhao, Li, Tong Xu, Xiawu Zheng, Enhong Chen, Rongrong Ji, and Xing Sun. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal 1lms in video analysis, 2024.
Noa Garcia, Mayu Otani, Chenhui Chu, and Yuta Nakashima. Knowit vqa: Answering knowledge- based questions about videos. In Proceedings of the Thirty- Fourth AAAI Conference on Artificial Intelligence, 2020.
11

---

Published as a conference paper at ICLR 2026
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yiamilos, Moritz Mueller-Freitag, Florian Hoppe, Christian Thurau, Ingo Bax, and Roland Memisevic. The "something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference Computer Vision (ICCV), Oct 2017.
Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, et al. Maniskill2: A unified benchmark for generalizable manipulation skills. arXiv preprint arXiv:2302. 04059, 2023.
Deepak Gupta, Kush Attal, and Dina Comner-Fushmant A dataset for medical instructional video classification and question answering. Scientific Data, 10(1):158, 2023.
Sanjay Haresh, Daniel Dijkman, Apratim Bhattacharyya, and Roland Memisevic. Clevrskills: Compositional language and visual reasoning in robotics. In NeurIPS, 2024.
Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang. Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos, 2025.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Yunfan Jiang, Agrim Gupta, Zichen Zhang, Guanzhi Wang, Yongqiang Dou, Yanjun Chen, Li Fei-Fei, Anima Anandkumar, Yuke Zhu, and Linxi Fan. Vima: General robot manipulation with multimodal prompts. arXiv preprint urXiv:2210.03000 2022.
Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In CVPR, 2024.
Alon Lavie and Abhaya Agarwal. METEOR: An automatic metric for MT evaluation with high levels of correlation with human judgments. In Chris Callison-Burch, Philipp Koehn, Cameron Shaw Fordyce, and Christof Monz (eds.), Proceedings of the Second Workshop on Statistical Machine Translation, pp. 228-2311 Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL httpt:t/:/// /tteer//allaaaahaaa Maathhoonooy org/ -W7-
Jie Lei, Licheng Yu, Mohit Bansal, and Tamara L Berg. Tvqa: Localized, compositional video question answering. arXiv preprint wexivv:809.0/11 009.01696, 2018.
KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang. Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, 2024a.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In CVPR, 2024b.
Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hierarchical encoder for video+ language reniesepresnntttionnn pre-training. arXiv preprint arXiv:2005.0 002000 2020.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Lid: An image is worth 2 tokens in large language models. In ECCV. Springer Nature Switzerland, 2024c.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 2023.
Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. BRLLhttp::/// http//acaannhooooooo org W04 1013/
Danni Liu, Gerasimos Spanakis, and Jan Nichues. Low-latency sequence-to- -o--uueee---seeeceeeeeee speech recog- nition and translation by partial hypothesis selection. In Helen Meng, Bo Xu, and Thomas Fang Zheng (eds.), INTERSPEECH, 2020.
12

---

Published as a conference paper at ICLR 2026
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang. Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning. ocr, and world knowledge, January 2024a. URL https
//1lava-vI.
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. TempCompass: Do video LLMs really understand videos? In ACL, August 2024b.
Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019, URL https
//axxiv.org//// abs wtt.://gg/ab//..111..... 1711105101111111
Xiaojian Ma, Silong Yong, Zilong Zheng, Qing Li, Yitao Liang. Song-Chun Zhu, and Siyuan Huang. Sqa3d: Situated question answering in 3d scenes, In ICLR, 2023.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-ChatGPT: Towards detailed video understanding via large vision and language models. In Lun- Mei Ku, Andre Martins, and Vivek Srikumar (eds.), ACL, August 2024.
Dominik Machácek, Raj Dabre, and Ondrej Bojar. Turning whisper into real-time transcription system. In IJCNLP-AACL 2023 System Demonstration, 2023.
Joanna Materzynska, Guillaume Berger, Ingo Bax, and Roland Memisevic, The jester dataset: A large-scale video dataset human gestures. In ICCV Workshops, Oct 2019.
Munan Ning, Bin Zhu, Yujia Xie, Bin Lin, Jiaxi Cui, Lu Yuan, Dongdong Chen, and Li Yuan. Video-bench: A comprehensive benchmark and toolkit for evaluating vidco-based large language models, 2023.
NVIDIA. Cosmos-Tokenizer: A suite of image and video neural tokenizers. https://githi:
com Comm/WWDDA/Cosmoss--ie
z 2025. Archived on 2025-02-11 accessed 2025-05-15.
NVIDIA, :, Niket Agarwal. Arslan Ali, Maciej Bala, Yogesh Balaji, Erik Barker, Tiffany Cai, Prithvijit Chattopadhyay, Yongxin Chen, Yin Cui, Yifan Ding, Daniel Dworakowski, Jiaojiao Fan, Michele Fenzi, Francesco Ferroni, Sanja Fidler, Dieter Fox, Songwei Ge, Yunhao Ge, Jinwei Gu, Siddharth Gururani, Ethan Hc, Jiahui Huang. Jacob Huffmann, Pooya Jannaty, Jingyi Jin, Seung Wook Kim, Gergely Klár, Grace Lam, Shiyi Lan, Laura Leal- -aixe Anqi Li, Zhaoshuo Li, Chen-Hsuan Lin, Tsung- Yi Lin, Huan Ling, Ming- Yu Liu, Xian Liu, Alice Luo, Qianli Ma, Hanzi Mao, Kaichun Mo, Arsalan Mousavian, Seungjun Nah, Sriharsha Niverty, David Page, Despoina Paschalidou, Zeeshan Patel, Lindsey Pavao, Morteza Ramezanali, Fitsum Reda, Xiaowei Ren, Vasanth Rao Naik Sabavat, Ed Schmerling, Stella Shi, Bartosz Stefaniak, Shitao Tang. Lyne Tchapmi, Przemek Tredak, Wei-Cheng Tseng, Jibin Varghese, Hao Wang, Haoxiang Wang, Heng Wang. Ting-Chun Wang, Fangyin Wei, Xinyue Wei, Jay Zhangjie Wu, Jiashu Xu, Wei Yang, Lin Ten-Chen, Xiaohui Zeng. Yu Zeng. Jing Zhang, Qinsheng Zhang, Yuxuan Zhang, Qingqing Zhao, and Artur Zolkowski. Cosmos world foundation model platform for physical ai, 2025. URL https::/arxxt org/ abs wwws//551.. 0tt.//2111.....7.
Sunny Panchal, Apratim Bhattacharyya, Guillaume Berger, Antoine Mercier, Cornelius Bohm, Florian Dietrichkeit, Reza Pourreza, Xuanlin Li, Pulkit Madan, Mingu Lee, Mark Todorovich, Ingo Bax, and Roland Memisevic. What to say and when to say Live fitness coaching testbed for situated interaction. In NeurlPS, 2024.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, 2002. Association for Computational Linguistics. doi: 10.3 10.3115/1073033
URL httt:s::/acllenthooooo..oogg 1040/
Viorica Patraucean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, joseph heyward, Mateusz Malinowski, Yi Yang. Carl Doersch, Tatiana Matejovi- cova, Yury Sulsky, Antoine Miech, Alexandre Fréchette, Hanna Klimezak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Simon Osindero. Dima Damen, Andrew Zisserman, and Joao Carreira. Perception test: A diagnostic benchmark for multimodal video models. In NewrIPS, 2023.
13

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via targe-scale weak supervision. In JCML, 2023.
Sana Javaid Raja, Adeel Zafar, and Aqsa Shoaib. A dataset for programming baaaad instructional video classification and question answering. In Proceedings of the First Workshop of Evaluation of Multi-Modal Generation, pp. 1-9, 2025.
Samyam Rajbhandari, Jeff' Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models, 2020. URL https: btttps:/aarxiii http:://xxiivvv.ov otge 1910 02054
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision-language model's perception the world at any resolution. arXiv preprint arXiv:2409. 2024.
Ruijie Wang. Zhiruo Zhang. Luca Rossetto, Florian Ruosch, and Abraham Bernstein. Nlgxform: A language model-basee question to spargi transformer, 2023. URL httlhttss///arxw..org
abs/ abs/2311. 2211. 22/1//2111..3333
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Ming Ding, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, and Jie Tang. Lvbench: An extreme long video understanding benchmark. pp. 22958-22967, October 2025.
Zeqing Wang, Wentao Wan, Qiqing Lao, Runmeng Chen, Minjie Lang, Xiao Wang, Feng Gao, Keze Wang, and Liang Lin. Towards top-down reasoning: An explainable multi-agent approach for visual question answering. 2026.
Anran Wu, Luwei Xiao, Xingjiao Wu, Shuwen Yang. Junjie Xu, Zisong Zhuang, Nian Xie, Cheng Jin, and Liang He. Deqa: Document-level chart question answering towards complex reasoning and common-sense understanding. 2023. Uttts:::: wwtp://wwwwg//a//////aabba
18983
Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos, 2024. URL /tRLltttpss://xxx..rrr/////
09711
Junbin Xiao, Angela Yao, Yicong Li, and Tat-Seng Chua. Can i trust your answer? visually grounded video question answering. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 13204-13211 2024. doi: 10. 10099//VR5273..222400000
Haiyang Xu, Qinghao Ye, Xuan Wu, Ming Yan, Yuan Miao, Jiabo Ye, Guohai Xu, Anwen Hu, Yaya Shi, Guangwei Xu, Chenliang Li, Qi Qian, Maofei Que, Ji Zhang, Xiao Zeng, and Fei Huang. Youku-mplug: 10 million large-scale chinese video-language dataset for pre-training and benchmarks. CoRR, abs/2306.04362. 2023.
Jin Xu, Zhifang Guo, Jinzheng Hc, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report, 2025. URLphtts: /ttpss://aaxiiii /awxiv..vvv. org/ abs//////////// 2503. 20215
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng. Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li. Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL https::/aaxxt
ww..or/..a.... att/ 2 2407. 10671
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin
14

---

Published as a conference paper at ICLR 2026
Yang. Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang. Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yugiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025a. URL https: htttttt/::/axax //arxiv org 2505. 09388
Rui Yang, Hanyang Chen, Junyu Zhang, Mark Zhao, Cheng Qian, Kangrui Wang, Qineng Wang. Teja Venkat Koripella, Marziyeh Movahedi, Manling Li, Heng Ji, Huan Zhang, and Tong Zhang. Embodiedbench: Comprehensive benchmarking multi-modal large language models for vision-driven embodied agents, 2025b. URL hattps://openreviert net. forum? DgGF2LEBPS
Qiaojun Yu, Ce Hao, Junbo Wang, Wenhai Liu, Liu Liu, Yao Mu, Yang You, Hengxu Yan, and Cewu Lu. Manipose: A comprehensive benchmark for pose-aware object manipulation in robotics. arXiv preprint wwxx:22403.13600 2024.
Amir Zadeh, Michael Chan, Paul Pu Liang, Edmund Tong, and Louis- Phiii-ppillppe Morency. Social-iq: A question answering benchmark for artificial social intelligence. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 8807-8817, 2019.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language org/ awaaaass23300 /asttss//aaxi...oz///b////333/3 wttp:///a/a//...////////
image pre-training, 2023.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding. 2025.
Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.0224 02858, 2023a. URL https://
Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory- real-time understanding for long video streams. arXiv preprint arXiv:2406. 0000:2406..8885 08085,
2024.
Hongjie Zhang, Yi Liu, Lu Dong, Yifei Huang, Zhen-Hua Ling, Yali Wang, Limin Wang, and Yu Qiao. Movqa: A benchmark of versatile preprint arXiv:2312. 048117, 2023b.
for long-form movie understanding. arXiv
Qiming Zhang, Jing Zhang, Yufei Xu, and Dacheng Tao. Vision transformer with quadrangle attention. arXiv preprint wxXiv:2303.55000 2023c.
Tianyi Zhang* Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In ICLR, 2020.
Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, al. Languagebind: Extending video-language pretraining to 2023.
modality by language-based semantic alignment. arXiv preprint
15

---

Published as a conference paper at ICLR 2026
Supplementary Material for QIVD
SUPPLEMENTARY CONTENTS
Social Impacts of QIVD
17
A.I Limitations of QIVD
17
A.2 Privacy and Ethics in QIVD
17
A.3
Broader Impact of QIVD
17
B
Additional Dataset Details
17
I
Additional Examples
17
B.2
Semantic Taxonomy
17
When-lo-Answer Statistics
18
B.4
Face-to-Face Samples
20
Additional Experiments
20
C.1
Comparing QID With Other Datasets
20
Correlation with Standard Metrics
Static Temporal Reasoning Gap
22
Category-wise Performance
22
C.5
Using Different Video Sampling strategies
22
LLM Judge Accuracy
Failure Cases
23
C.8
Statistical Significance
24
D Additional Experimental Details
24
D.I
Implementation Details
24
D.2
VideoLLaMA Finetuning Details
25
D.3 Stream-Qwen-Omm Details
25
D.4
LMM Evaluation
27
D.5
GPT-4o Prompt
27
D.6 GPT-4o Refusal Cases
27
16

---

tests higher-level action interpretation and Published as a conference paper at ICLR 2026
A SOCIAL IMPACTS OF QIVD
A.1 LIMITATIONS QIVD
While our experiments with QIVD indicate that it presents significant challenge for current multi- modal models-and despite the dataset being human-validated for annotation accuracy- there are several potential limitations and sources of bias consider:
1. Relatively small class sizes, which may limit the diversity of questions and answers; 2. Recordings conducted in controlled environments, potentially reducing variability in lighting, background, and camera angles, which may affect model generalization; 3. Possible demographic biases in terms of gender, age, and ethnicity, which could impact model performance across diverse user groups.
A.2 PRIVACY AND ETHICS QIVD
The data was collected under direct agreements with crowd workers, permitting both research and commercial use, ensuring compliance with applicable privacy regulations, including GDPR- equivalent standards. All videos were manually reviewed identify and exclude any content containing issues such as the presence of individuals in the background. Personally identifiable information, including metadata, was removed to the extent possible to ensure participant privacy. Additionally, all contributors received fair and appropriate compensation in accordance with the standards of their respective regions. All contributors signed consent form that explicitly permits both research and commercial use of their video and audio data, including use in training Al models. We will provide a contact email on the dataset release page, allowing participants to request data removal at any time.
A.3 BROADER IMPACT QIVD
In addition to the aforementioned sources of bias, language models trained QIVD may generate harmful or biased content, propagate misinformation, or offer inappropriate advice. These risks must be carefully considered when interacting with, deploying, building upon such models.
B ADDITIONAL. DATASET DETAILS
B. ADDITIONAL EXAMPLES
We show additional video examples from our dataset in Figure B.l demonstrate the diversity of examples in QIVD.
B.2 SEMANTIC TAXONOMY
A detailed definition of the categories used in QIVD is provided here.
Action Attribute: Inquiries regarding the manner in which an action was performed, such as Which hand did use wave? How fast did jump?- tests ability to recognize fine-grained characteristics of dynamic events.
Action Counting: Questions about the frequency of action's repetition, such as How many times did I clap? evaluates temporal reasoning and event segmentation capabilities.
Identifying the specific action that performed, such What am doing right now?-asseser basic activity recognition in dynamic scenes.
Action Detection:
Action Understanding: Questions about the purpose outcome of action, such as What does this gesture mean? or Why am I moving the intention recognition.
Object Attributes: Inquiries about the characteristics of an object, such as What color this
book? ? or Is this cup empty or full? evaluaesl finengrained visual perception of static properties.
17

---

Published as a conference paper at ICLR 2026
Figure B.1: 1: Each image showcases different video from our collection, demonstrating the substantial variation in visual scenarios captured within the dataset. These examples highlight the diversity of environments (indoor and outdoor settings), participants, objects, actions, lighting conditions, camera angles, and compositional elements present across the dataset.
Object Counting:
Determining the number of objects present, such as How many pens are on the table? ? tests quantitative reasoning and object individuation.
Object Detection:
Identifying an object within the scene, such as Is there a lamp in this room?
assesses basic object recognition capabilities.
Object Referencing: Indirectly pointing to an object within the scene, such as What am pointing at? or What is behind me? evaluates spatial reasoning and deictic reference resolution.
Object Understanding: Questions about the nature or function of an object, such as What is this tool used for? semantic knowledge about objects beyond mere recognition.
Inquiries about the environment, such as What room am I in? or Is it daytime or nighttime? evaalutee holistic scene interpretation.
Scene Understanding:
Audio-Visual: Questions that require audio information for a complete answer, such as What sound am making? or Am speaking loudly or softly-tests tests cross-modal integration capabilities.
OCR: Extracting text from an object, such as What does this sign evalueval the capability to recognize text in the real world and within the context of the conversation.
Subjective: Soliciting general opinions about an object or scene, such as Does this outfit look good?-test tests a model's s ability to respond sensibly subjective questions.
B.3 WHEN-TO-ANSWER STATISTICS
Figure B.2 plots, for every clip, the temporal offset between the moment an answer becomes valid and the end of the vidco. Most questions answerable within the last quarter of the clip, yet the long, asymmetric tail indicates that a non-trivial fraction require substantially earlier or later responses, confirming the need for models reason the full temporal span rather than assume a fixed "answer now" point.
We also quantify how often the correct answer becomes valid only after the question has finished. Because ground-trutt truth end-of-question timestamps are unavailable, we use the of--qusttion de- tected by Streaming-Whisper Whisper proxy. Table B. reports the count and fraction of clips whose
18

---

Published as a conference paper at ICLR 2026
Probability
102s
Median: 100s
0.15
Dev: 0.74g
0.10
0.05
4
Time Difference (8)
Figure B.2: Distribution of optimal answer time relative the end of each clip. A value of on the horizontal axis means the ground-truth "answer now" moment occurs x seconds before the video finishes.
Table B.I: Time difference statistics between ASR end-of-question and ground-truth when-to-answer.
Time Threshold
Count
Percentage
0.0 after
2054
100.0%
0.5 after
1351
65.8%
1.0 after
745
36.3%
2 1.5 after
375
18.3%
2.0 after
186
9.1%
2 2.5 after
90
4.4%
2 3,0 after
MI 4.0 after
1.7%
5
0.2%
ground-truth when-to-answer time occurs at least given threshold after the ASR end-of-question time.
Figure B.3 shows the empirical distribution At, the error of our streaming-ASB estimate relative to the human wwhen-to-answer annotation. The skew toward negative values reveals systematic tendency of the ASR system to answer questions prematurely, often before sufficient visual context is available. Together, the figures highlight both the variability of optimal answer timing in real- col-world interactions and the practical challenge of detecting that moment reliably in a streaming setting.
Probability
0.30
Mean: -0.55s
0.25
Median: -0.445
Std Dev: 0.99s
0.20
0.05
3
-1
5
At (s)
Figure B.3: Histogram of the signed error At = tASR foT between the streaming-ASR trigger and the human "answer now" annotation.
19

---

Published as a conference paper at ICLR 2026
Table B.2: Facial expression and gesture recognition samples.
Question
Answer
Do I look tired?
You seem tired.
What I' doing right now?
You smiling.
Is this monkey happy or sad?
The monkey on the pillow appears to be happy.
Am happy?
It appears that you are happy.
What expression is this?
An expression of sadness or disappointment.
Did approve or disapprove?
You gave thumbs up, indicating your approval.
Am smiling?
Yes, you smiling.
Am cold?
It seems that you are feeling cold.
Am thinking?
It seems like you are deep in thought.
Did it hurt?
Yes based on your expression it did hurt.
Am approving?
Yes, its seems like you giving your approval with the thumbs up gesture.
Did sleep enough last night?
You do not look sleep deprived.
Am smiling?
No, you' re not smiling.
What action is this?
You appear to be afraid of something.
What am I doing?
You are raising your arms showing victory or happiness gesture.
Table B.3: Evaluation of representative baseline LMMs facial expression and gesture recognition
samples of QIVD. Corr. represents the correctness score calculated by the LLM judge.
Model
Corr. f
BERT
METEOR 1
BLEU f
ROUGE-L
Xuu al.
2025
33.90
86.33
25.16
0.51
12.57
Qwen2.5-VL. VL-BB Wang el al.
2024
44.07
86.78
29.35
0.83
17.23
Owen3-VL-B8 Yang el al.
2025a
61.02
86.81
28.49
0.73
19.7
Demmini-2.5-Flas (C Comamci et al.
2025
33.90
88.74
32.10
3.32
21.05
GPT-4o Hurst et 2024
32.20
85.48
36.98
2.44
21.28
B.4 FACE-TO-FAAE SAMPLES
Understanding facial expressions and gestures a critical capability for any personal assistant system. To illustrate that QIVD effectively incorporates this functionality, we identified all samples within QIVD that are directly or indirectly associated with facial expression interpretation and gesture detection. In total, these comprise 59 samples, distributed as follows: 4 from action counting, 20 from action detection, 20 from action understanding, 10 from object attributes, 2 from object referencing, from scene understanding, and 1 from subjective categories. A curated set of representative examples is presented in Table Furthermore, we evaluate the performance of representative models on these samples, using ground truth questions and timestamps. The corresponding results are summarized in Table B.3
C ADDITIONAL EXPERIMENTS
C.1 COMPARING QIVD WITH OTHER DATASETS
We examine whether QIVD overlaps visually semantically with prior video-QA corpora by embedding every clip with the Cosmos-CV8x88 tokenizer (NVIDIA 2025 NVIDIA et al. 2025) and measuring distances to clips from the closest public datasets: AVSD (DSTC7) (Alamri al. 2018), and Social-IQ (Zadeh et al. 2019). For each video, we first normalize the frame-rate to 8 FPS and truncate (or zero-pad) to a maximum of 64 frames (8 seconds). The input frames are resized to 224 224 pixels. The resulting tensor, after batching and permuting
V E d xpxxxx44x22x (Batch x Channels x Time x Height x Width) format, is passed through the tokenizer's 's encode method. This yields continuous latent representation. We average these latent features across the temporal and spatial dimensions (dimensions 2, 3, and 4), obtaining a single embedding vector e per clip.
Table C. reports, for each clip in QIVD, the L2 distance to its nearest neighbour within QIVD itself well as AVSD (Alamri et al. 2018) and Social-IQ (Zadeh et 2019). The intra-QIVD distances
20

---

Published as a conference paper at ICLR 2026
Table C.I: Nearest-neighbor L2 distances between QIVD clips and each dataset.
Comparison
Mean
Median
Min
Max
5th Percentile
QIVD vs. QIVD
0.0157
0.0148
0.0031
0.1124
0.0062
QIVD vs. AVSD
0.0386
0.0369
0.0125
0. 0.1238
0.0173
QIVD vs. Social-IQ
0.0894
0.0871
0.0257
0.2156
0.0458
SS Moustostion
Q./
A253
Socaal Q
-15
Figure C.I: lio-dimensional t-SNE projection of the 1024-dimensional embeddings for QIVD (blue), AVSD (red), and Social-IQ (green). QIVD clips form tight, coherent cluster that is clearly separated from those AVSD and Social-IQ, illustrating their distinct distributions in latent space.
(mean 0.0157,5t 5th percentile 0.0062) are smaller than the inter-dataset distances: AVSD (mean 0.0386, 5th percentile 0.0173) and Social-IQ (mean 0.0894, 5th percentile 0.0458). Only small fraction of QIVD clips find their closest counterpart outside the test split indicating minimal overlap with prior benchmarks and underscoring that QIVD brings substantially novel visual semantic content.
Figure C.I further illustrates this separation in a two-dimensional t-SNE projection of the 0124-iima embeddings: QIVD points form a tight cluster on the right, clearly distinct from AVSD (red) and Social-IQ (green), which occupy disjoint regions. We demonstrate that QIVD significantly different from prior datasets.
C.2 CORRELATION WITH STANDARD METRICS
In Table C
report the correlation between standard metrics (BLEU, ROUGE-L, METEOR,
BERTScorC) and the LLM Judge correctness score. We observe that ROUGE-L (r = 0.774) and METEOR = 0.678) correlate relatively well with the judge, while BERTScore (r 0.616) shows a lower correlation.
Table C.2: Correlation between standard metries (BLEU, ROUGE-L, METEOR, BERTScore) and the LLM Judge Correctness score across all evaluated models.
Metric
Pearson (r)
Spearman (p)
BLEU-4
0.675
0.680
ROUGE-L
0.774
0.737
METEOR
0.678
0.688
BERTScore
0.616
0.506
21

---

Published as a conference paper at ICLR 2026
Table C.3: Correctness (%) comparison between Static (Object-related) and Temporal (Action/Scene- related) categories. Humans maintain consistency, whereas models show sharp decline in temporal tasks.
Model
Static Categories
Temporal Categories
Gap
Human
88.44
86.46
-1.98
GPT-4o
65.24
46.06
-19.18
Gemini-2.5-FFaas
66.10
44.14
-21.96
Swen3-VL-88.
67.06
46.36
-20.70
Qwen2.5-VL-
57.69
36.97
-20.72
VideoL.LaMAA37B
62.40
45.25
-17.15
Static Tasks
Temporal Tasks
Correctness (%)
100
80
88.4
86.5
65.2
67.1
62.4
40
46.4
Human
GPT-4o
Qwen3-VL
VideoLLaMA3
Comparison model performance on Static vs. Temporal tasks. While humans perform equally well on both, all models show significant performance degradation temporal tasks.
Figure C.2:
C.3 STATIC VS. TEMPORAL REASONING GAP
analyze the performance gap between "Static" categories (Object Attributes, Counting, Detection, Referencing, Understanding, OCR) and "Temporal" categories (Action Attributes, Counting, De- tection, Understanding, Audio- Visual, Scene Understanding). Table C.3 reveals striking disparity: while human performance remains consistent across both types 1-87-88%) all ttate-ofthh--as art models suffer a significant performance drop (>> 19%) when moving from static to temporal reasoning tasks. We observe that current L.MSs biased towards static image capabilities and struggle with the dynamic nature of real-world video interactions.
C.4
CATEGORY-WISE PERFORMANCE
Table C.4 provides the granular correctness scores for each semantic category across the top- performing models. We observe that even the best models achieve 35% on some categories, such as Action Counting, compared to 85.7% for humans.
C.5 USING DIFFERENT VIDEO SAMPLING STRATEGIES
We study how restricting visual evidence to short temporal window around the moment the question is spoken affects performance. For given clip, we consider the segment spanning +0.5, +100, +2.0. +3.0, +4.0, or =:5.0 seconds around the question timestamp and uniformly sample frames from that segment. We evaluate Qwen2.5-VL-7B under these settings well as the full-video baseline. Results are reported in Table C.3 Consistent with the intuition that many questions require context before and after the utterance, very short windows harm performance. Wider windows recover accuracy, with +4 seconds yielding the best results; beyond 55 seconds, inputs effectively cover the full clip and performance plateaus.
22

---

Published as a conference paper at ICLR 2026
Table C.4: Detailed breakdown of Correctness (%) by category for top-performing models and Human baseline (Offline/GT setting).
Calegory
GPT-4o
Gemini
Qwen3-VL
Qwenn255-V
LaMss
Action Attributes
83.33
50.97
54.84
50.97
46.45
47.10
Action Counting
85.71
7.59
20.09
20.09
11.16
33.48
Action Delection
88.64
60.00
49.77
52.95
42.73
48.18
Action Understanding
85.71
59.09
50.00
62.73
48.
47.27
Object Attributes
78.33
71.71
70.05
77.05
64.77
71.35
Object Counting
96.67
Object Detection
95,24
68.72
70.62
62.09
30.71
66.35
Object Referencing
89.87
56.37
61.53
61.76
55.10
60.20
Object Understanding
100,00
64.56
59.49
68.35
56.96
54,43
Scene Understanding
75.00
78.95
73.68
73.68
73.68
73.68
Audio-Visue
100.00
4.35
21.74
21.74
0.00
34,78
OCR
100.00
69.57
78.26
73.91
60.87
47.83
Subjective
60.00
69.77
30.23
72.09
51.16
Table C.5: Effect of sampling frames within temporal windows around the question timestamp for
Qwen2.5-VL-7B (Wang et al. 2024). Values are proportions or similarity scores (higher is better).
Model
Corr. f
BERT
METBOR
BLEU
ROUGE-L
Qwen2.5-VI--7B (full video)
60.00
87.58
37.37
4.66
Qwen2.5-VL-7B (sampled around question) 10.5
38.28
78.90
25.40
2.60
20,30
Qwen2.5-VL-7B (sampled around question) $1.0s
42.00
80.10
28.90
3.10
Qwen2.5-VL-77 (sampled around question) 1
60.41
87.00
42.02
4.60
Qwen2.5-VL-7B (sampled around question) 1
61.20
87.90
38.10
4.72
Owen2.5-VI.-7B (sampled around question) $
60.75
88.01
37,80
4.68
29.55
DW--2 (sampled around question) 5
59.95
87.72
37.55
4.64
29.48
C.6 LLM JUDGE ACCURACY
To evaluate the accuracy of the LLM judge, we randomly select subset of 300 samples from QIVDand collect human ratings of GPT-4o's (Hurst al. 2024) answers. In addition to the human evaluation, we use three automatic judges: LLaMA3-8B (et. al. 2024) and two recent Qwen3 (Yang al. 2025a) models (32B and 8B). The fraction of answers deemed correct by each evaluator reported in Table Based on Table we use Owen3-8B (Yang et al. (025a) as the main judge
throughout our experiments. We provide the results with LLaMA3-8B (et. al. 2024) as the judge in O
Table C.6: Correctness GPT-4o answers on a 300-sample subset under different evaluators. Values the proportion marked correct.
Evaluator
Correctness
Human Evaluation
0.64
LLaMA3-8B (et. al. 2024h
0.68
Qwen3-32B YYang et 2025a
0.57
Qwen3-8B (Yang et al.
2025a
0.59
C.7 FAILURE CASES
To further demonstrate the limitations of current LMMs in addressing routine real-life questions, we present a series of simple queries that, while effortlessly answered by human annotators, pose significant challenges for LMMs (see Figure C3 Notably, these examples highlight the shortcomings of several advanced models, including the robust GPT-4o, the large-scale WdcoLLaMA2-11 (Zhang
et al. 2023a), and even the fine-tuned
(Zhang et al. 2023a
23

---

Published as a conference paper at ICLR 2026
Table C.7: Evaluation of baseline LMMs the QIVD dataset using (a) questions and estimated when-to-answer timestamps by Whisper (Radford et al. (023) and (b) ground-truth questions and timestamps. Corr. represents correctness by LLM judge with LLaMA3-8B (et. al. 2024) as the judge.
ASR Questions and Timestamps METBOR
Human Questions and Timestamps
Model
Corr:
BERT
BLEU
ROUGE-L
Coer.
BERT"
METEOR
BLEU
ROUGE-L
Chat UsiVi
37.47
6.08
18.45
90.50
40.00
7.24
31,22
InstmueDD
82.19
4.35
41.16
92.03
10.72
LLIMA-VID
PE 20235
43.45
91.51
37.18
5.54
29.50
18.48
91.77
31.55
5.42
29.82
LLaVA NcX1
24,97
85.29
22.35
1.38
11.64
28.90
85.78
24.50
1,67
13.22
Maure
2014
55.38
90.53
38.
7.58
31
40.76
91.01
40.50
9.07
39.58
ViaeCChat
Li
5.00
85.15
23.48
1.08
12.22
8.31
85.30
24.30
1.03
12.54
VideoChar har
1eari pirth
46.07
91.13
45 45.49
11:35
41 11.38
53.07
91.52
47.90
12.43
43.67
Vide-1Lla Vla Zamala 2003 et 1023
23.52
87.77
27.15
1.38
19.31
18.62
83
2.50
0.00
15.06
VideoLLMMA
PUP44)
33.52
89.50
29.05
7.62
30,54
39.21
90.45
43.58
9.86
34,93
LaMe
1.ceee111 2004
64.31
91.18
47.30
13:93
10.63
$2.64
91.71
$1.08
16.41
43.97
Videol
67.89
91.42
46,60
14.04
41.31
53.41
92.29
51,13
16.12
45.76
VadooLLaMA1TT
52331
90.92
45.20
11.21
40.55
59.62
91.63
48.56
12.72
43.84
VideoLLM caline
76.60
27.36
2.51
20.39
38.45
33.08
3.99
25.26
Faali-VStmaam
Phim
8985
28.95
4.17
37 07.05
90.48
31,49
5.05
29.90
Qwan25-VL177
53.55
87.17
34.95
3.80
26.52
60.00
87.58
37.33
4.66
29.44
Qwen2.5
44.76
85.65
33:45
20.57
66.9%
86.73
33.98
08
Qwml-VI-6B 68
87.08
33.90
31.53
64.60
87.58
36,73
35.89
Gemini-2 S.llor
2025
66.41
90.41
43.07
8.33
36.05
GPT [ars 2022
15.72
Human (sibset)
89.35
93011
53.21
17,40
49.76
Table C.8: ASR performance comparison.
Model
METEOR
BLEU
ROUGE-L f
L
Whisper (Radford et al. 2023)
90.01 23.11
80.95 + 35.13
90.32 + 22.66
Chisser-Streaming (Macháeek
92.34 = 15.31
74.57 = 33.52
91.82 + 15.72
0.83 + 0.77
et al.
2023)
C.8
STATISTICAL SIGNIFICANCE
We report the standard deviation values corresponding to table table and table 5 table C.8 table C.9 and table C.10
Table C.9: Evaluation of baseline LMMs on the QIVD dataset using questions and when-to-ae -nnwwer timestamps extracted by "hisper-Streaming (Radford et al.) 2023). Corr. represents the correctness score calculated by the L.M judge.
Model
Cor:. 1
BERT
METEOR 7
BLBU
ROUGE-L
Char-UniVi Jin er al. 2024
34,66 (1:47.60)
89.94
(4:3.56)
37.47 (1-23.53
6.08 (=16.44)
28.45 (:::22.41)
(1.77)2)
(:10.3)
(1.14.40)
LID Lret 20246
39.41 (1:48887)
90.51 (:13.56)
37.18 (1:23.25))
5.84 (116.99)
29.80 (::22.003)
LLaVA-NeXT Lau 2024
(::39.59)
85.29
(1..688)
(::15.211
Video-ChanGP
(Maa el A024
32.45 (::44688)
90.53 (1:3.78)
38.1 (::24.78)
7,58 (219.46)
31.09 (::244.55)
VideoChat i
3.69 18.85)
85.05 (:12.777)
23.48 (:1:15.29)
1.08 (:16.47)
12.22 (:1:12.2)
VideoChut2 pl 20246
44,66 (:1.4972)
91.13 (.13.88)
45.49 (.1.26.63)
11.35 (3.23338))
41.38 26.04)
Video-LLaVa
2023
2023
20,28 (1::40.21)
87.77 (::3,37)
27.15 (:::18.88)
1.98 (1:9.73)
19.31 (::17763)
VideoLLaMA Zhang al 20160
30.76 (.1.4.16)
89.50 (.14..6))
39.05 (.1.2..0))
7.62 18.87)
30.84 24.83)
VideoLLaMAZ-75
2024
43.34 (::59.66)
91.18 (::4.118)
47.20 (:1:27.92)
13,93 (6226)))
40.63 (:::27.22)
VideolLaMAA-7255 728 (iimnns
46.52 (1:49989)
91.42 (::5.66)
(4:228.88)
14.04 (=127.41)
41.71 (=:28.50))
Videal aMA3-7B
50,59 (1:50.01)
90.92 (::5.33)
45 20 (1::27.14))
11.21 (3:23.54)
40.55 (c:26.55))
(1.38.40)
(.1.2.111)
19.30)
Flash- VSreeam ZZhing et al. 2024)
44.28 (:1:99
89.8
(::33373)
28.95 (:1:24.21)
4.17 15.38)
27.05 (:::44.56)
Qwen2.5 V-L-78
(9.75)
(::23.25)
Qwen2.5-Omma
Xxir 75tl
43.97 (9.64)))
86.65 (1:1
33.45 (1:17.12))
2.77 (::5.94)
20.57 (2:122711)
Qwen3-VI..c VL.
Yang
53.72 49.87)
87.08 (1.0088)
33.90 (1:22.11)
5.29 (2:72.70
31.53 (.127.10))
D ADDITIONAL EXPERIMENTAL DETAILS
D.1 IMPLEMENTATION DETAILS
All experiments were conducted in PyTorch. Every open-source LMM checkpoint was loaded in half-precision (FP16), except for the 72B parameter Videol.LaMAs (Zhang al. 2023a model,
24

---

Published as a conference paper at ICLR 2026
Question: way point? GPT-40: You should point your left.
Question: What is the color of the mug in my left hand?
GPT-4o: The color the mug in your left hand black.
x
VideoLLaMMA
point, you should extend your your index finger indicate specifie direction object.
Lade2L
mug your left hand black.
x
x
VideoLLaMA FT:
pointing your left.
VidcoLLaMAAY FT: The mug your left band is black.
X
x
Human: You're pointing the right. You're using your hold hand with your elbow to your side and palm facing
Human: white.
Action Attributes
Object Altributes
Question:
How many did say ah?
Question:
look tired?
GPT-4o: Based the provided video frames. possible determine bow many times said "sh".
GPT-4o: gullt zisilaggng
x
appearmon accervily
VideoLLaMM2 72 B: You said "ah" lotal of 12 times the
VideoLLaMA2 72 B: No, look great.
video.
x
VideoLLaMA.A - FT: You said "ah" three times.
VideoLLaMAAV FT: No. you don't look tired.
x
Humann One time.
Human; You seem tired.
Audio Visual
Object Attributes
Question: How many times do I hend?
Question: Did hurt?
GPT-4o: Based on the provided video frames. you pal your bead once.
GPT-4:: Booice vivee
but bust indivitaaii bodh prrctingg snd por dhvomatenn
x
Imgag:
VideoLLaMA2- 72 B: times
VideoLLLaMAZ 72 B: was not too bad.
x
VideoLLaMA.N AV FT: You patted your head twice.
VideoLLaMA FT: No, it not hurt.
x
x
Human: You pat your head four times.
Human: on your expression
Action Counting
Action Understanding
Figure C.3: Simple daily face-to-face questions that strong baseline LMMs such as GPT-4o, VideoLLaMA2-728, and LadeoL 1-7--AV fail to answer.
which was run with post-training INT8 quantization to satisfy memory limits. The inference code for each baseline was taken unmodified from the authors' public repositories and executed with the best-performing
settings provided by the authors. All of our experiments were run on single A100-80 GB GPU.
D.2 VIDEOLLAMA FINETUNING DETAILS
We initialize from the publicly released VideoL.LaMA 2.1-7B-AV (Zhang al.) (023a) checkpoint and reuse the authors' training recipe with minimal modifications. The 2900 clips in QIVD are partitioned into 5 non-overlapping folds via deterministic hash of the video filename. Each fold in turn serves validation, while the remaining four constitute the training split (~2.32K clips). The architecture components updated during fine-tuning are summarized in Table D.I hyperparameters in Table D.2
We present all the
D.3 STREAM- LOENN-QEE--E OWEN-OMNI DETAILS
This section provides details how convert the Qwen2.5- O0men..5-Omni model to streaming format. The conversion is achieved by changing the way multi-modal data is provided to the model. We split the audio-visual data into I-second chunks and feed the model one chunk at time. The model is
25

---

Published as a conference paper at ICLR 2026
Table 1.10: Evaluation of baseline LMMs on the QIVD dataset using ground-truth questions and timestamps. Corr. represents correctness by LLM judge.
Model
Cor.
BERT
METEOR
BLBUT
ROUGE-L
Char-UnnVVi Jin
2024
40.79 (1:49.15)
90.50 (1.3.49)
40.02 (1:23:64)
7.24 (8.229)
31.22 (::222.0))
InstretDLLIP
Dar et 2025
39.14 (1.4888)
82.03 (1.3.13)
4.54 (1.6.8)
0.07 (1.1.70)
10.72
14.56)
LLaMA-VID
20216
43,00 (1:49.52)
90.78 (2:3.32)
37.55 (1:22.42)
5.42 (51,,9))
29.82 (:::21.12)
LLaVA-NeXT
L-u cl 2
20218
22.66 (+:41.87)
85.78 (2:3.44)
24.50 (4:16.66)
1.67 (#:9.53)
13.22 (::16.54)
Video-ChinGi1
[Mary ot 6124
36,59 (2:48.18)
91.01 (2:3.78)
40.59 (4:255.0))
9.07 (=21.51)
33.58 (::255.11)
3,52
(1:15.51)
(L.5.5))
(1.12111
VideoCha12
Li 31246
50.34 50.00)
91.52 (1:3.81)
47.93 (:1266.62)
12.43 (4:24.04)
43.87 (::25..77)
Video-LLLaV Zhi
(1.38.1)
83.38
2,90 (1:5.27)
(:10.00)
(1.06.00)
Vide Zhana i. 20266
35,93 (1:44.99)
90.45 (1:4.15)
43.88 (1:255.81)
9.86 (:221.99)
34,93 (5.25.09)
VidkoLLaMAZ-155 &ienees
(=:28.88
(::277.66)
Videol.LMM22772M
(2 Fi Mure
30.83 (0:50.00)
92.29 (2:4.35)
51.13 (#:277.5))
16.12 (= 86)
45.76 (8.28.06)
(2inn
56.38 (:::49.60)
91.6 63 (1:4.24)
48.56 (1:26681)
12.72 (4224.22
43.84 (::26-11))
Videol. M-nnine Chen et 2024
23.62 42.442
88 45 (.L3.55)
13.08 (11.21.2))
3,99 12.35)
25.26 19.97)
Flash-VStream Zhhing
49.59 (0.50)1
90.48 (::3.57)
31.49 (:1:24.88)
5.05 (2)11
29,90 (:1:24.98)
Dowen2.5-VL-715 Waigg M0 2024
50.62 (+:50.0)
87 (222.63)
37.37 (4:20044)
4,66 (=: 11.67)
29.44 (::24.18)
Doven2.5-0mni22 (h
4025
45,90 (2:44.84)
86.73 (2:1.93)
33.98 (1:17..2))
2.87 (:L5.96)
20.98 (2:12.71)
Qwen3-VL. (Yani 20A60
87.58
(1.22.77
14.11)
28.07)
Gemini 2.5-Fladi (Comancc et al. 2025
58.07 (:1:49.35)
90.43 (::4.12)
43.07 (:1:25.20)
8.33 (0220.)8)
36.05 (:1:66.01)
GPT-4o Hurst
(::49.24)
89. 15.25)
(d:277.32)
(=22.27)
(::28817)
(1:25.22)
(:125.18)
Table D.1: Trainable versus frozen modules during fine-tuning.
Module
Status
Vision encoder
(Zhai et al. 2023
Frozen
Audio tower (BEATs iter-3 Chen et al. 2022)
AS-2M)
Trainable
Multimodal projector (A)
Trainable
Multimodal projector (V)
Frozen
LLM backbone (Qwen2-7B-Instruct Yang et al. 2024
Trainable
fine-tuned generate a special token ."...) when it is listening and watching, and to produce an answer when it reaches the time-to-answer.
For this purpose, we reformat the training data shown in the following example.
Original format:
"L "Lime_to Lo_aanwweerta "3.3s" "answer": holding Rubik's cube your hand."
Streaming format:
[0.0a, 1.0s. ....",,
[1.0s, 2.0s. (2.0s, 3.0s. ....").
[3.0s, 4.0s, [4.0s, 5.0s, "..
holding Rubik/s cube your left hand."],
[5.0s, 6.0s,
schematic of this architectural modification is shown in Figure D.I Stream-Qwen-Ome model detect when-to-ansswr with granularity of one second.
This approach enables the
During finetuning, all weights are kept frozen except for the vision adapter, audio adapter, and embedding layer. The network is trained with batch size of 1, gradient accumulation steps of 1, and 2 full-size frames per second for epoch. Other training details follow Table D.2] Similar to Videol LaMA finetuning, we 5-fold cross validation with the same splits.
26

---

Published as a conference paper at ICLR 2026
Table D.2: Hyper-parameters and optimization settings for each volid-aalio fold.
Hyper-parameter
Value
Training precision
bf16
Global batch size
8 videos (1 x 8)
Frames per clip Epochs
8
2
Optimizer
AdamW (Loshchilov Hutter 2019)
Adam (31, $2, e)
(0.9, 0.999, 10
Weight decay
Learning rate schedule
10 - (cosine), 3% warm-up
Gradient accumulation steps
8
Gradient clip-norm
1.0
Distributed strategy
Deepspeed ZeRO-2 (parameter off-load)
Rajjbhandari et al. 2020)
A
Streaming Qwen-Omni
Offline Qwen-Omni
1s
1s
1s
Multimedia
Multimedia
Best response time
Best response time
Figure D.1: Details
Stream Stream-Qwen-Onl Qwen-Omni structure.
D.4 LMM EVALUATION
The prompts supplied to the LLM-based judge reproduced in Table D.3 and Table D.4 Since subjective questions require a qualitatively different notion of correctness, we evaluate those cases with dedicated prompt that deems responses acceptable provided they are contextually appropriate, friendly, and affirmatively phrased.
D.5 GPT-40 PROMPT
To process QIVD videos with GPT-4o, we uniformly select four frames from each video and spatially downscale them to half their original size. The preprocessed frames are then combined with the question into query, as illustrated in Table D.5 and this query used prompt GPT-4o.
D.6 GPT-40 REFUSAL CASES
GPT-4o declines answer 76 questions in QIVD due sesponssibeeATPo111 cyVie ew alt i onn. Given that the samples in QIVD undergo extensive quality checks, the likelihood of samples violating the ResponsibleAAPolice samples as Responsibles iibilblee wwwesisslleePPPll... www.lleAAPocccyvill... cyViolat ion and refuses to provide an answer. We consider these cases, where GPT-4o provides empty response, incorrect in our evaluations. Examples of questions that GPT-4o refused to answer are shown in Figure D.2
is very low. In these instances, GPT-4o mistakenly classifies the
27

---

Published as a conference paper at ICLR 2026
General Correctness
Evaluation
You inteligentt chathot that is unmatched world expert evaluating the factual accuracy generative outputs for video-based question answer pairs. You are tasked with evaltating the curreciness of predicted answer by comparing it reference answer. answers are to the same question. You perfactly compare predicted answers the reference answer and determine if they are factually consistent. As needed, you expertly consider the short version of the reference answer which contains only relevant details, and the question category.
System Prompt:
You perfectionist adhering these criteria for correctness: Follow these sleps:
You are given the Question, the Category, the Reference Answer (short), the Reference Answer, and the Predicted Answer.
Read the Question: Carefully read and understand the question provided.
Read the Category: Take note of the category the question understand the context.
Read Reference Answer (sbert): Carefully read understand the reference short that contains key
point.
the short answer 'NA', IGNORE the short answer
Read the Reference Answer: Carefully read and understand the reference answer provided.
Read the Predicted Answer: Carefully read and understand the predicsed answer that beeds to be evaluated.
Compare the Statements: Compare the predicted answer to the reference answer, focusing on the accuracy of the information and the presence of key details. Pay VERY CLOSE attention to the following notes:
predicted answer directly addresses question and aligns with the reference is key
tion.
Verify that the predicted answer does contradict the reference answer.
for logical consistency
question
predicted answer.
The reference answer or the predicted answer may include extra details that are not requested the question.
Only consider the answer details relevant the question.
The predicted answer
factually accurate and consistent with reference answer.
Consider sy synonyms or paraphrases as valid matches.
If the predicted answer refusal to unswer, treal it as INCORRECT:
Provide a Judgment: Baxed on your comparison make decision if the predicted answer is CORRECT or INCORRECT.
User Prompt: Please evaluate the following video-based question-answer pair:
Question: [Question]
Question catlggor: (Question category]
Reference Answer: (Relerence Answer]
Reference Answer (short): |Reference Answer (short))
Prodicted Answer: Predicted Answer]
Provide your evaluation only as a score for the predicted answer where the score is for INCORRECT and for CORRECT.
Generate response form Python dictionary string with single key 'score'. and its value the factual accuracy score as an INTEGER
DO NOT PROVIDE ANY OTHER OUTPLT TEXT OR EXPLANATION AND DO NOT RETURN INVALID DICTIONARIES, Only provide the Python dietionary string.
For example, your response should look like this: scoorr 1nt (score)
Table D.3: We use these prompts to evaluate the correctness LMM-generated answers.
28

---

Published as a conference paper at ICLR 2026
Subjective Correctness Evaluation
You are an intelligent chatbot that is an unmatched world expert evaluating the factual accuracy generative outputs video-based question answer pairs. perfectly compare predicted they are faciually consisteni. As needed. experily consider the short version of the reference unswer which contains only relevant details, and the question category. Since the question subjoctive, you treat answers that are contextually relevant, friendly, and ideally include details from reference reference
System Prompt:
reference answer determine
CORRECT.
You perfectionist achering these additional criteria for INSTRUCTIONS
Compare the predicted unswer to the relerence answer and short reference answer.
If the predicted answer is positive, lriendly, and includes details from the reference unswer, it is CORRECT:
If predicted answer is blank, it is INCORRECT.
If predicted
refusal answer,
INCORRECT. HOWEVER, if the reference also claims
it is not possible and this matches the predicted answer, it is CORRECT.
If predicted
does include details bui responds affirmative such "Yeah' "That cool!',
AND a sensible answer to the question, it is CORRECT.
The predicted answer should NOT contain misinterpretations misinformation.
The reference
may include details relevant to the question.
requested the question. Only consider the details
Consider synonyms paraphrases as valid matches.
If short reference answer is 'NA, IGNORE the short answer.
User Prompt: Pleuse evaluate the following video based question answer puir:
Question: [Question]
Reference Answer: [Reference Answer]
Reference (short): |Reference (short)
Predicted Answer: [Predicted Answer]
Provide your evaluntion only as a score for predicted unswer where the score for INCORRECT and for CORRECT.
Generate the response in the form of Python dictionary string with single 'scare", und its value the factual accurncy score as an INTEGER.
PROVIDE ANY OTHER OUTPUT TEXT EXPLANATION AND DO NOT RETURN INVALID DICTIONARIES, Only provide the Python dictionary string
For example, response should look like this: ('sooref int (score)
Table D.4: We use these prompts evaluate the correctness LMM- AMM-generated answers.
Question: How many times did clap?
GPT-4o: I'm sorry. I cannot assist with that request.
x
Ground Truth: You clapped total of three times.
Question: How many times jumped!
GPT-46: I'm sorry, cannot assist with that request.
Ground Truth: You jumped twice.
Question: Is this monkey happy or sad?
GPT-4o: I'm sorry, cannot assist with that request.
Ground Truth: The monkey on the pillow appears to be
happy.
Question: What am wearing?
GPT-4o: I'm sorry, cannot ussist with that request.
x
Ground Truth: just put pair of glasses.
Figure D.2: Examples of questions that ResponsibletrPooloyyytoo..l
GPT-4o refused to answer due to
29

---

Published as a conference paper at ICLR 2026
GPT-4o prompt
messages
"role": "aystem",
"costent"; "You expert analysis. question using what happening in the video frames."
"zole": "user",
"contens":
ttpe"" "text",
"eaxt" "faaa""""aased on provided frames, lquestion]
ttpe"" "image_uri", image_uri";
"url": s"daas:inoee/jjeg/ahsee (execdee__fram__L]] "detail"; "high"
Lype": "image_uci", image_uri";
"uri": tmage/st base64, (er fercoddd_ffamm "detail"; "high"
"sype": "image_url", imoge_uri":
"uzl": "ddcaccingge/jeeeybeeee (encoded_[rame__]) "detail": "high"
type"" "image_uri", image_ur1";
"url"; daars:imoge//jpeghhaeett (encoded__frmee "detail"; "high"
Table D.5: The prompt used to run inference with GPT-4o.
30


Reviews:
[{"review_id": "VpbaAT89f8", "reviewer": "Reviewer_KsFd", "summary": "The authors introduce the Interactive Video Dataset (IVD), a new video QA benchmark for evaluating the ability of multimodal large language models (MLLMs) to answer face-to-face questions in real-time. The authors provide a comprehensive evaluation of several baseline MLLMs on IVD, finding that current models significantly underperform compared to human-level performance.", "questions": "- Why is the streaming evaluation protocol (Sec 5.1) needed? It relies on Whisper for timestamps, which is not measuring the tested model's own capability to process the information and decide when to answer in real-time. This external dependency adds noise from Whisper's performance, and it seems like using the human annotated timestamps makes more sense. A more interesting setup would be asking the model itself to provide a timestamp of when-to-answer, and compare that with the annotated ground truth.\n- The key distinction of IVD compared to prior real-time QAs seems to be the face-to-face element (Table 1). Why is face-to-face important for real-time QA? Intuitively, a face-to-face setting implies a model should infer user intent from non-verbal cues like facial expressions or gestures, but this does not seem to be the focus of IVD. Could the authors elaborate on what makes the \"face-to-face\" aspect a unique and necessary contribution beyond what existing real-time datasets offer?\n- The authors claim that IVD poses a \"strong demand on situational context understanding\" compared to post-hoc created real-time datasets.  Could the authors expand on this claim, particularly in comparison to post-hoc created datasets like VStream-QA? One could argue that datasets like VStream-QA, with their longer context and less restrictive (face-to-face) settings, offer an equally or even more realistic real-time evaluation setup.\n- While not a weakness, the paper would be strengthened by including experiments on generalization of the IVD-finetuned model -- does a model fine-tuned on IVD show improved performance on other offline face-to-face datasets or other online streaming video benchmarks?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The contribution of a real-time, interaction-focused video QA dataset is timely, addressing a growing interest in multimodal AI assistants and real-world interaction.\n- The collection of videos in \"in-the-wild,\" realistic settings is a strength, ensuring the dataset's quality, diversity, and practical relevance\n- The authors show that models fine-tuned on IVD achieve improved performance, confirming the dataset's value as a resource for advancing model capabilities in this domain.", "weaknesses": "- The video clips are generally short (~5 seconds), limiting the dataset's demand on a model's contextual memory which is an important capability for real-world real-time assistants.\n- While the paper evaluates an extensive list of models, they are often generic MLLMs not designed specifically for real-time use cases. Evaluating baselines developed specifically for real-time QA, such as FlashVStream, would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NW0UktbGMw", "reviewer": "Reviewer_SQFL", "summary": "This paper introduces, IVD, a comprehensive benchmark, and dataset designed to assess and train LMMs on a wide variety of tasks requiring responding to humans in real time. Besides, the authors also propose a streaming baseline approach that couples a streaming Automatic Speech Recognition (ASR) model to detect the end of a question with a Video-LMM to generate an answer based on the preceding context. Experiments show that existing models fall far behind human performance on this task.", "questions": "The work lacks technical novelty. The core flaw is assuming optimal response timing equals question end time, which avoids the actual research challenge. The dataset is small, evaluations miss recent models like Qwen3-VL and GPT-5, and the paper lacks theoretical depth in analyzing model failures.", "limitations": "", "rating": 2, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The fundamental idea of this paper is technically correct.\n2. This work shifts focus from offline video analysis to the challenges of real-time, situated interaction, which is critical for the future of human-AI collaboration.\n3. The paper is easy to follow.\n4. The rationale behind each section and the overall motivation are clearly presented and easy to understand.", "weaknesses": "1. While the authors introduce a valuable dataset and problem formulation, the technical contribution is limited in novelty. The core technical contribution is merely combining a streaming ASR system (Streaming-Whisper) with existing Video-LMMs. This approach lacks innovation, essentially representing a straightforward concatenation of existing components without novel architectural design or algorithmic improvements.\n3. The paper's core weakness is its reliance on the overly simplistic assumption that the optimal time to answer is the end of the spoken question. Although the authors acknowledge this limitation, basing their entire method on such a fundamentally flawed indicator introduces significant error and sidesteps the actual research challenge of learning when to respond.\n3. The dataset contains only 2,900 videos, which is relatively small.\n4. The experimental evaluation could be strengthened by including more recent and powerful models. The current comparisons feel incomplete as they omit several state-of-the-art models that have been released, such as Qwen3-VL and GPT-5.\n5. The paper lacks deep theoretical analysis of why existing models perform poorly on this task. While failure modes are identified, there is insufficient exploration of root causes and no targeted solutions are proposed for the fundamental issues.\n6. A very interesting and counter-intuitive finding is that the pre-trained VideoLLaMA model's performance degraded when audio was included. Could the authors provide a deeper analysis or hypothesis for this phenomenon?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "K149EGr6Ly", "reviewer": "Reviewer_2VbA", "summary": "The paper proposes a new dataset, the Interactive Video Dataset (IVD), to evaluate the online responsiveness of LMMs.", "questions": "1. For the Streaming-Whisper row in Table 4, when calculating METEOR, BLEU, and ROUGE-L for transcribed text, is the calculation based on the full transcribed text or the transcribed text up to the \"when-to-answer\" timestamp identified by the model?\n2. For VideoLLaMA2.1-7B-AV, after adding audio information, why did model performance conversely decrease? Are there any reasonable explanations for this phenomenon?\n3. Regarding the \"Impact of when-to-answer\" setup in Section 5.1, the study compares model performance when using ground-truth and ASR-derived when-to-answer timestamps. All models in Table 5 could participate in this comparison, so why is Qwen2.5-Omni specifically emphasized? With the same ASR transcription model used, even if the LMM itself does not support speech transcription, it should not affect the study on the \"Impact of when-to-answer.\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses an important issue for AI systems: how to answer face-to-face questions. By providing a new dataset, it standardizes this problem, which helps inspire the research community to focus on the online responsiveness of LMMs.", "weaknesses": "1. The paper solely uses the Streaming-Whisper model to determine \"when-to-answer.\" Whether this leads to insufficient accuracy in evaluating the performance of large models in answering face-to-face questions remains questionable. In particular, the paper notes that \"the end of a question does not necessarily capture the optimal moment for an answer.\" Is it necessary to introduce more suitable models for determining \"when-to-answer\"—for example, models trained specifically for question-answering timing, rather than simply detecting the end of a question?\n2. In face-to-face scenarios, a key consideration is that AI agents actually receive continuous video streams. These streams have no definite start time and can be regarded as infinitely long. The streaming approach proposed in the paper focuses on determining \"when-to-answer,\" but identifying \"when the current discussion begins\" is equally important (even without considering complex multi-turn dialogues). In the current dataset, it appears that multimodal information from the start of the video up to the \"when-to-answer\" timestamp is defaulted to input into the LMM. This is not feasible in real video streams.\n3. The scale of training data used for fine-tuning is relatively small.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "duELWPDTF8", "reviewer": "Reviewer_vNTF", "summary": "This paper introduces IVD (Interactive Video Dataset) — a new benchmark designed to evaluate vision-language models (VLMs) in realistic face-to-face question answering scenarios. Unlike existing VideoQA datasets, IVD emphasizes “when-to-answer” (i.e., temporal readiness), audio-visual reasoning, and deictic references that commonly appear in human–AI interactive settings.", "questions": "- Is there any plan to open-source this dataset in the future? \n- Will there be more scenarios for real-time (face-to-face) question answering？\nGood job, no other questions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Novel and realistic task formulation\n- The paper highlights a crucial but underexplored problem — when and how VLMs should answer in interactive real-world scenarios.\n- This setup moves beyond static VideoQA and aligns better with embodied AI and multimodal assistant applications.\n\nHigh-quality dataset design\n- IVD provides rich annotations: question semantics, answer timestamps, and multimodal data (audio + video).\n- The inclusion of “best-answer time” is innovative, enabling quantitative evaluation of temporal readiness.\n\nComprehensive evaluation\n- The study benchmarks several leading LMMs (VideoLLaMA2.1, Qwen2.5-VL, GPT-4o) under both zero-shot and fine-tuned settings.\n- The analysis spans multiple aspects: when-to-answer accuracy, ASR quality, audio contribution, and fine-tuning gains.\n\nInsightful analysis\n- The t-SNE visualization of 1024-D embeddings effectively illustrates the domain shift between IVD and traditional VideoQA datasets.\n- The paper provides qualitative failure cases and detailed error categorization (e.g., deictic confusion, premature answering).\n\nRelevance and potential impact\n- The benchmark fills an important evaluation gap for real-world multimodal dialogue and will likely spur further research in streaming and interactive LMMs.", "weaknesses": "Limited dataset scale and diversity\n- Only \\~2.9K samples with short video clips (\\~5 s).\n- The data collection is crowd-sourced and potentially biased toward controlled indoor scenes, limiting generalization.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)