Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders
Authors: Ali Rasekh
Erfan Bagheri Soula
Omid Daliran
Simon Gottschalk
Mohsen Fayyaz
Venue: NeurIPS 2025 poster ()
Keywords: Video-LLM
Video Understanding
Temporal Reasoning
Video Question Answering
Large Vision Language Model
Multimodal LLMs
Primary area: deep_learning
TL;DR: We introduce our Video-LLM architecture, STAVEQ2, with stacked temporal attention in the vision encoder, significantly improving fine-grained temporal understanding in video question answering task.

Abstract: Despite significant advances in Multimodal Large Language Models (MLLMs), understanding complex temporal dynamics in videos remains a major challenge. Our experiments show that current Video Large Language Model (Video-LLM) architectures have critical limitations in temporal understanding, struggling with tasks that require detailed comprehension of action sequences and temporal progression. In this work, we propose a Video-LLM architecture that introduces stacked temporal attention modules directly within the vision encoder. This design incorporates a temporal attention in vision encoder, enabling the model to better capture the progression of actions and the relationships between frames before passing visual tokens to the LLM. Our results show that this approach significantly improves temporal reasoning and outperforms existing models in video question answering tasks, specifically in action recognition. We improve on benchmarks including VITATECS, MVBench, and Video-MME by up to +5.5%. By enhancing the vision encoder with temporal structure, we address a critical gap in video understanding for Video-LLMs. Project page and code are available at: https://alirasekh.github.io/STAVEQ2/

Enhancing Temporal Understanding in Video-LLMs through Stacked Temporal Attention in Vision Encoders
Ali Rasekh
Erfan Bagheri Soula* Independent Researcher erf an. b. soula@gmail. com
Omid Daliran"
Leibniz University Hannover,
Independent Researcher
L3S Research Center ali risaraheehess.co
hopebraves@gmail. com
Simon Gottschalk
Mohsen Fayyaz
Leibniz University Hannover,
Microsoft
L3S Research Center
mobsenffayrarmmmicrrotttttccm
gottschalk@L.3..d
Abstract
Despite significant advances in Multimodal Large Language Models (MLLMs), understanding complex temporal dynamics videos remains a major challenge. Our experiments show that current Video Large Language Model (Video-LLM) architectures have critical limitations in temporal understanding, struggling with tasks that require detailed comprehension of action sequences and temporal pro- gression. In this work, we propose a Video-L.LM architecture that introduces stacked temporal attention modules directly within the vision encoder. This de- sign incorporates temporal attention in vision encoder, enabling the model to better capture the progression of actions and the relationships between frames before passing visual tokens to the LLM. Our results show that this approach significantly improves temporal reasoning and outperforms existing models video question answering tasks, specifically in action recognition. We improve on benchmarks including VITATECS, MVBench, and Video-MME by up to +5.5%. By enhancing the vision encoder with temporal structure, we address critical gap in video understanding for Video-LLMs. Project page and code are available https://alirasela. Brrich..gttta gitb....o.. io/ /0//TAVEQ2///////
Introduction
Recent advances in Multimodal Large Language Models (MLLMs) have led to significant improve- ments performance across a wide range of tasks involving multimodal data, including video question answering and image captioning demonstrating impressive capabilities in integrating both visual and textual information. However, despite these advancements, when comes to videos, current Video Large Language Models (Video-LLMs) still face major challenges in understanding temporal dynamics.
In video question answering (VQA), the input is a video accompanied by natural language question, and the output a textual answer to that question. While current models perform reasonably well on spatially grounded questions (e.g.. "What color is the ball?"), they struggle when the query requires precise comprehension temporal progression within the video. Notably, many existing benchmarks
Denotes equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Video:
Video:
Question: happening video?
Question: is happening in video? -Options (GT Action: Pulling from right to left)
(GT Action: Biking)
Qwen2-VL: PL: A man is seen biking, he is wearing a T-shirt.
Qwen2-VL- Pulling something from left to right
STAVEQ2:
Someone is riding a bicycle, moving from right to left while wearing red shirt.
STAVEQ2: Pulling something from right left
Figure I: Responses from Owen2-VL. and our STAVEQ2. Left: For temporally simple action (Bik- ing), both models answer correctly, Right: For a temporally challenging action (pulling something from right to left), Qwen2-VL provides incorrect answer, while our STAVEQ2 succeeds.
and datasets are not particularly challenging in terms of temporal complexity, allowing some models answer questions by relying a single frame. In contrast, tasks such action recognition necessitate understanding not only frame content but also how the frames change over time. An example is given in Figure where the action captured in the second video is identifiable only when observing at least two frames.
In this work, first analyze the limitations of current Video-LLMS such Qwen2-VI VL 44 and Intern Video2-Chat [47 in temporal modeling capabilities. Specifically, we observe that current Video-LLMs struggle with fine-grained temporal reasoning tasks, such distinguishing between actions that differ subtly in their execution over time (e.g., pulling an object from left to right versus right to left). Furthermore, despite attempts improve performance through in-context learning approaches, these models consistently fail to recognize similar temporal patterns across different video instances, indicating fundamental limitation in their temporal processing architecture rather than merely training data issue. Building upon our findings in the limitations of the current tate-offthhe-rt models, we propose our novel video-LLM model STAVEQ2.
With STAVEQ2, we propose an enhanced Video-LLM architecture that introduces stacked temporal attention modules directly within the vision encoder. As detailed in Figure this architectural change explicitly equips a Video-LLM's vision encoder with temporal attention, enabling it to better capture the progression and dynamics of actions across frames before passing the visual tokens the LLM for final reasoning. In our evaluation, we demonstrate that this design significantly improves the model's 's temporal understanding, leading superior performance temporally challenging tasks and benchmarks.
In summary, our contributions include: (i) We analyze how current Video-LLMs struggle in capturing complex temporal dynamics, even with in-context examples and fine-tuning. (ii) We propose STAVEQ2, our Video-LLM model, equipped with improved temporal video understanding. We show that STAVEQ2 outperforms recent state-of-the-ar Video-LLMs on several benchmarks. To the best of our knowledge, with STAVEQ2, blocks into the vision encoder of Video-LLMs for video question answering. (iii) We achieve new state-of-the-at results the SSv2 action recognition benchmark by applying our proposed temporal attention mechanism to previous state-of-the-art video foundation model (Vision-only).
the first to efficiently include dedicated temporal attention
2 Related Work
Video-LLMs
The field video understanding has witnessed significant advancements with the
emergence of Video-LLMss. Multimodal language models, which integrate additional modalities like into language models, have expanded the scope of L.MSs beyond text 12 Extending this the video domain, recent Video-LLMs have enabled tasks such as
images 42 20 or audio

---

video captioning, question answering, and instruction following by aligning video content with natural language. Models like
[3), Video-LLaMAf3], Video-LLaVA [0], LLAVA-NeXT-
Video[54] VideoChat [27]. Otter 26)] Gpt4Video 49 and Intern VI. 2.5 g demonstrate strong performance in grounding text visual inputs. While they effectively capture spatial semantics, many still struggle with modeling long-range temporal dependencies and maintaining consistency across frames. This challenge is underscored in some works (2), which show case that even state-of-the-an video-language models lack inherent understanding of temporal order, and proposes additional temporal information is necessary to improve these lacks. Our work builds on these models by focusing on improving temporal understanding in video-based reasoning tasks.
Temporal Understanding in Video-LLMs Despite the advancements in Video-LLMs, temporal understanding still an area that has not been fully explored, and unlike claims provoked by Liu et al. [32], existing models appear to struggle with understanding temporally complex tasks. Several specifically targeting very long videos [4] 40 24 I8]. Also, some works have been trying to push temporal understanding
works have made efforts to address this, such as those by 25 35 22
on models using methods such as temporal localization or boundary 6 35 22]. Others use Q-
formers to condition the temporal features inputs in order to improve temporal understanding and performance [36] M. TG- Vid [2I] introduces a time gating mechanism aimed at enhancing temporal modeling: however, its results remain modest and the method is not particularly efficient.
Prior works employ varied approaches to temporal modeling. For instance, Owen2-VL. relies solely on spatial attention in its vision encoder, delegating temporal understanding to the language model. In contrast, Intern Video2 incorporates joint spatiotemporal attention in its vision encoder. However, our experiments in the next section demonstrate that both approaches are insufficient for capturing fine-grained motion dynamics and temporal dependencies. However, another approach that has been less explored in Video-IIMMs is the divided space-time attention introduced in [5. Our experiments also explore multimodal in-context learning VLMMs, as studied in [16 revealing that existing models lack sufficient in-context learning capability.
3 Problem Definition & Motivation
Before introducing our model, begin by analyzing the performance limitations of current Video- LLMs on temporally demanding video question answering (VQA) tasks. Through a series of experiments, we reveal fundamental weaknesses in temporal reasoning, which motivate the need for our proposed approach.
Problem Definition We address the task of video question answering (VQA), where the input consists of a video V E IR H x3 with T' RGB frames of height H and width W, and natural- language question Q. The goal is to produce a textual answer that accurately responds to the question based on the visual content of the video. This formulation generalizes to variety of tasks such as video captioning, action recogmition, and similarity detection. For example, action recognition can be cast as a VQA task by posing the question: "What action is happening in this video?" with an answer such as "Pulling [something] from left to right." " Our focus is on temporally challenging VQA tasks, where the correct answer depends not static frames, but modeling the sequence and progression of visual elements over time. For instance, distinguishing an action such as moving an object from left right versus from right left requires precise temporal reasoning.
We create a temporally challenging VQA dataset, specifically curated to stress temporal understanding. We evaluate the performance of selected Video-LLMs (Qwen2-VL, Intern Siremniide2--Cie and LLa VA- NeXT NeXT-Video; see Section 2 on this dataset under zero-shot settings. Then, we investigate whether performance be improved via conventional LLM improvement approaches such as through in-context learning. Finally, we analyze the results to demonstrate the inherently temporal nature of dataset and the persistent limitations of current models in handling such tasks.
3.1 Temporally Challenging Dataset
Our experiments utilize the Something SoaiiiinnnSomtiingggggoo v2 (SSv2) dataset 17 benchmark with over 220K videos, each annotated with one of 174 actions (e.g.. "Pulling [something] from left to right"). To examine temporal challenges, select a subset of SSv2 with action classes
34 an action recognition
3

---

Table 1: Zero-shot and in-context action recognition performance of four Video-LLMs. We use different LLM to judge whether the generated answer matches the ground-trus -ouuh action (LLM-as-a- Judge). *e provide the possible actions within the prompt.
Qwen2-VL 2B
Qwen2-VL.
Intern Video2
LLaVA NeXT-Vides 7B
# Examples
7B
Chat 8B
14.87%
21.91%
30.60%
19.38%
0
24.01 %
35.91%
46.11%
31,46%
1
9.24%
15.83%
23.86%
16.54%
3 5
9.56%
16.79%
17.04%
18.98%
8.92%
20.72%
10.41%
16.32%
that are opposites in how they change over time. This way, create the dataset SSv2-T10, that specifically targets at remporally-challenging cases by reducing SSv2 to 14,462 videos 10 classes that represent pairwise counterparts regarding their temporal characteristics. These classes were chosen to create temporally challenging VQA tasks, where accurate action recognition depends on modeling the sequence and progression of visual elements across frames, aligning with goal of evaluating Video-LLMMs temporal reasoning capabilities. The list selected action classes is provided in Appendix B
3.2 Zero-Shot & In-Context Performance
We analyze whether the performance of the tested Video-LL.Ms can be improved through few-shot prompting, motivated by findings in multimodal LLMs showing that in-context learning enhances performance []4] (2) As shown in Table D they perform poorly in the pure zero-shot setting with accuracies as low as 14.87% for Qwen2-V VL 2B and only 30.60% for Intern Video2-Chat. Even when we provide the list of 10 candidate classes within the prompt, performance remains weak. Although scores improve notably this setting (e.g.. 46. 46% for Intern Sideo2-Chatt and 35.91% % for Qwen2- VL 7B), the models still often fail to select the correct class, despite knowing the task and being given the exact set possible answers. This indicates a limited understanding of the temporal aspects of video content. Such weakness is critical, one of the most important in video-based tasks namely, the direction and progression of motion- is not reliably captured.
To improve performance, provide examples of vidcos and their actions in the respective prompts (see Appendix[A for example prompts). However, the second part of deemonttetssssst that this strategy does not help. In fact, all models show performance drops examples added. Notably, all models except Qwen2-VL 7B drop in performance when the number examples increase These results suggest that current video-language models struggle with in-context learning for VQA, failing to recognize actions across video instances even when given explicit examples. We provide more experiments on different settings in Appendix F
3.3 Temporally Challenging VQA
In this experiment, we focus Intern Video2-Chat, as it achieves the best zero-shot performance among the evaluated models-likely due to its spatiotemporal attention mechanisms in the vision encoder. Figures 2a and prepeett confusion matrices for Video-iideo-Char on SSv2-T10, 000 treating action recognition as classification task. Without fine-tuning, the model struggles to distinguish be- tween directional actions such "Pulling [something] from left to right" versus "Pulling [something] from right to left", which are different their temporal aspects. After fine-tuning on SSv2-T10, its performance improves, but significant confusion between temporally mirrored actions remains, still fails to reliably distinguish actions that have different temporal meanings. These are actions that involve the same objects, actions and context and are visually very similar, but differ only in their temporal direction or order.
This result highlights a critical limitation: although Intern Video2- -hhat is architecturally better equipped for temporal reasoning than other models, still fails to reliably model directional informa- tion, fundamental aspect of video understanding. We attribute this to the lack of dedicated temporal

---

361
89
299
385
80
True labal
86
215
1
True label
305
109
200
3-
True
19
2
277
10
16
252
1
17
255
Predicted label
Predicted label
Predicted label
(a) Zero-shot.
(b) Fine-tuned on SSv2-T10.
(c) Fine-tuned with stacked tempo- ral attention.
Figure 2: Confusion matrices of Intern Video2-Chai performing action recognition on SSv2-T10 showing results the following classes: (1) Pulling [something] from left to right: (2) Pulling [something] from right to left; (3) Throwing [something] in the air and catching it; (4) Throwing [something] in the air and letting fall; (5) [Something] falling like rock.
modeling blocks in current Video-LLMs which hinders their ability reason about fine- ane-grained temporal structures, even when spatiotemporal cues are present in the architecture. As shown in Figure 2c after adding stacked temporal blocks, we can see that the model performs much better and is able to distinguish the actions accurately. This improvement confirms the importance of explicit temporal modeling in Video-LLMs. Corresponding quantitative results of this experiment are
provided in Appendix D (Table 8)
Model: STAVEQ2
Through our initial experiments in the previous section, we demonstrated the difficulties of state-of- the-art Video-LLMs to deal with temporally challenging VQA tasks, when in-context examples were provided. To address these challenges, we introduce our approach for enhancing temporal understanding in Video-LLMss in Video-LIMs: STAVEQ2 Stacked Temporal Attention in Visual Encoders for Qwen2-VL.
4.1 Preliminaries
MILMMs typically combine a vision encoder with LLM to process visual and textual inputs [31] The vision encoder extracts features from visual inputs (e.g., images or videos), which are then transformed into token embeddings via trainable adapters. These embeddings are fed into the LLM alongside textual inputs, enabling joint visual and linguistic understanding. For Video-LLMs. the vision encoder must effectively capture both spatial and temporal information to model the dynamics of video sequences.
Given video V, following the vision transformer paradigm [13], a vision encoder typically divides each frame into N HW/ p2 non-overlapping patches size P. These patches are projected into an embedding space, yielding a sequence of patch embeddings (0) RTxNx where x E IR denotes the embedding of the i-th patch in frame t, and D is the embedding dimension. Each transformer block the vision encoder applies multi-head self-attention across the patches within each frame, followed by feew-forward multilayer perceptron (MLP), producing spatially informed feature representations.
4.2 Overview
Our experiments show that relying entirely on the LLM to interpret temporal relationships between frames as by Qwen2-VL. is insufficient. Even the use of joint spatiotemporal attention models like Intern Video2- cannot adequately resolve this issue, as shown in our temporal analysis (Section 3.3). Furthermore, Intern Video2-Chat architecturally constrained to processing 8 input frames, which limits its applicability to vhoro tasks and precludes its use longer video
5

---

Answer A
*Pulling something from right to left'
Large Language Model
RoPE
Fraie 1+1
(m)
Question
Projector
X (M)
MLP
Frame
"What action happening in this video?'
LayerNorm
Vision Encoder
Frame t-1
Temporal Attention
Legend
X (0)
XM
Frazen
Patch Embed
LayerNorm
Fine-Tune (LoRA)
Spatial Attention
Full Train
LayerNorm
Framno
Query
Key
X (m-1)
Video V
Our proposed STAVEQ2 architecture. Video frames are processed through transformer blocks with spatial and stacked temporal attention modules, capturing intra-frand frame and inter-frame dynamics. The resulting visual tokens are fed into the LLM for answer generation.
Figure 3:
benchmarks. Therefore, a more flexible Video-LLMM must be developed effectively process both temporal and spatial information.
With STAVEQ2, propose enhancing the Qwen2-VL Video-LLM by incorporating temporal attention blocks. As illustrated in Figure 3 STAVEQ2 builds upon the standard vision encoder architecture, by adding dedicated temporal attention mechanisms after spatial attention blocks. This design generates token embeddings enriched with spatiotemporal information, which are aligned with the LLM through projector module for autoregressive answer generation in VQA tasks.
4.3 Stacked Temporal Attention in Vision Encoders
Each transformer block m, processes patch embeddings (m--1) RTxNND where X x((m-1 E RD represents the embedding the i-th patch (i i..... N)) in frame (t I,..., T') of the input video V. The block consists of spatial self-attention, injected temporal self-attention, and a feed-forward multilayer perceptron (MLP), with residual connections and layer normalization (LN) applied each stage. For clarity, we describe the formulation for single attention head. For spatial attention, queries, keys, and values are computed for each frame as:
(m)) K!
(((m) LN xm( w-(m) LN (x)m-1) W(())
LN (x(m-1)) w(m), (1)
where X E NxD is the embedding matrix for frame f, and W=(m) are learnable projection matrices, with d, as the dimension of queries and keys. The spatial attention weights are:
QQ K w(m) E Dxd,
Q((m) K (mm)
A softmax
/dg
and the output is:
(m) A (m) (((m)))))))) + x (m--1)
(3)

---

The spatial attention outputs are concatenated along the temporal dimension as S(m) [s(m], ..... 1 RTxNNx
For temporal attention, we process the spatial features for each patch across frames, defined as
yr(m) [sfm ......... PT.
s(myT E IRTxD The temporal attention mechanism computes:
Q km(m) V((m) LN (y:!m)
W LN (y:(m) m LN (y!!!) W
where W
Wxm) w(m)) E IR Dxdi are learnable projection matrices, and de is the dimension of queries and keys. The temporal attention weights are:
(g
K K((m)² (m))
A 4(m) softmax
(5)
Vd.
and the output is:
z(m) A A((m) v(lmm) y:(m)
(6)
The temporal attention outputs are concatenated along the spatial dimension as 7(m)
(m) N ( RxNxxx The block concludes with MLP and residual connection:
x MLP (LN (z(m)) (())
The feature representation (M) of the video V output from the last transformer block M is projected the LLM's embedding space, concatenated with the language embeddings of the question Q, and passed to the LLM. This setup allows the LLM to reason jointly over visual and linguistic modalities while benefiting from temporally-awe aware visual embeddings.
A LLM (Projector (x())) 2)
Our key innovation is the parameter-efffcient temporal attention module. By using up to four times fewer attention heads than spatial attention while maintaining the head dimension, we significantly reduce parameters (See Appendix [foor the ablation studies). We apply ID rotary position embed- dings (RoPE) in the temporal attention block to encode temporal structure, unlike the 2D RoPE used for spatial position encoding in models like Qwen2-VL. This lightweight design enhances temporal modeling with minimal computational overhead, improving the quality of video features for temporally challenging VQA tasks.
Evaluation
We evaluate STAVEQ2 several video understanding benchmarks, demonstrating that STAVEQ2 outperforms recent state-of-the-art Video-LLMs. We compare STAYEQ2 with the Qwen2-Vi 2B/7B/72B and Qwen2.5-VL 7B/72B [3], as well as other state-of-the-art models. Furthermore, by applying our proposed stacked temporal attention to the Intern Video2 video foundation model, achieve new state-of-the-art result on the SSv2 action recognition dataset. We also conduct experiments on a variation of the SSv2 dataset and try to see how well models can learn to match similarities between videos and to see how STAVEQ2 can compare. The experiments are conducted using 64 NVIDIA A100 GPUs.
5.1 Training STAVEQ2
To train our STAVEQ2 model, we use two-stage process that integrates the injected temporal attention blocks with minimal disruption to the pre-trained Owen2-VL. We train on VQA datasets using cross-entropy loss to optimize the generated textual answers for temporally challenging tasks. To train STAVEQ2 while preserving the instruction-followee models, we curate subset of Web Vid [ a large-scale video-caption dataset, and generate multi-turn question-answer pairs by prompting the Qwen2 7B [51] LLM. We refer to this dataset WebVid-QA.
capabilities of base instruction-tunel
7

---

Table 2: Performance for fine-tuning on SSv2 (full dataset). Our Video2 1B + Stacked Temporal Attention (STA) sets new saate-of-th--asee art on the full SSv2 dataset, outperforming the larger Intern 6B model by 0.5%.
Model
Accuracy (%)
VideoMAE V2-H
43
76,8
VideoMAF V2-g
43
77.0
MVD-L 45
76.7
MVD-H 45
77.3
InternVideol IB 47
77.1
Intern Video |4
77.2
Intern Video2 6B 47
77.5
Intern Video2 IB STA
78.0 (1 0.5%)
In the first stage, we initialize the output projection layer of the temporal multi-head attention blocks to zero, ensuring the vision encoder initially behaves like the original model and preserves its pre- trained spatial modeling capabilities. We freeze all model parameters except the temporal attention blocks and the associated layer normalizations. These are trained with linear warmup over the first steps, gradually integrating temporal attention into the encoder.
In the second stage, introduce LoRA adapters [9] the linear layers of both the vision encoder (attention projections and MLP) and the LLM, using small rank to maintain parameter efficiency, The temporal attention blocks and LoRA adapters are jointly trained to align the enhanced spatiotem- poral features with the LLM's linguistic reasoning. This stage ensures that the model effectively processes temporal dynamics for VQA tasks, enhancing feature quality for the LLM, as validated by its performance on temporally challenging questions.
5.2 Selected Benchmarks
We evaluate our method on multiple tasks and benchmarks, including full SSv2 action recognition, SSV2-VSM dataset for visual similarity matching, and three diverse video understanding benchmarks, each testing different aspects of video comprehension: VITATECS [29] diagnostic dataset disentangled from static information for temporal concept understanding of Video-LLMs across six aspects: compositionality, direction, intensity, localization, sequence, and type. MVBench [28]: A multimodal video understanding benchmark covering 20 challenging video tasks that cannot be effectively solved with a single frame. Video-MME [5]: A comprehensive benchmark for evaluating MLLMs video understanding across 6 primary visual domains and various durations.
5.3 Temporal Performance Improvements
To analyze the temporal modeling capabilities of stacked temporal attention module, we enhanced the Vitern vision-only model with STA and fine-tuned it the full SSv2 dataset for action recognition. This experiment showcases STA's ability to boost temporal understanding, even in models with pre-existing joint spatiotemporal attention. The results, detailed in Table reveal that Intern Video2 IB with STA achieves new ttate-of-the- accuracy of 78.0%. outperforming the larger Intern Video2 6B model by 0.5%, despite Intern Video2 1B STA having only about 1.3B parameters. This leap in performance underscores the efficiency of STA's dedicated temporal focus, proving that the gains arise from enhanced temporal modeling rather than just modest parameter increase. Comparisons with other models are also presented in Table
5.4 Visual Similarity Understanding
To further investigate the temporal understanding of Video-LLMs, we also examine their ability to perform visual similarity matching. Since transformer-basee models operate using self-attention, detecting similarity between tokens is a relatively straightforward task for them. Therefore, the vision encoder is effectively capturing and representing video content, the model should be able to

---

Table Evaluation results for fine-tuning on SSv2-VSM, The task is to decide which of two videos is similar to a third video.
Model
Accuracy (%)
Qwen2-VL
68.65
STAVEQ2 2B
72.19 (1 3.5%)
Qwen2-VIL LL 7B
73.15
STAVEQ2 7B
76.05 (1 2.9%)
Table 4: Accuracy (%) on video understanding benchmarks for our STAVEQ2 compared to other models. For VITATECS, aspect-wise results are shown; other benchmarks report overall accuracy. *( "(Video-MME without/with subtitles). & Results collected from the Video-MME leaderboard. indicates results not reported the original paper and unavailable from other sources.
VITATECS
Model
MVBench
* *VMME (wo/w)
Comp.
Dir.
Int.
Loc. Seq.
Type
Qwen2-VL 2B
80.8
82.1
69.6
76.1
72.2
85.9
63,2
55.6 / 60.4 56.2 / 61.3
STAVEQ2 2B (Ours)
81.3
83.0
70.1
76,9
72.9 86.6
65.1
ST-LLM 7B
54.9
TGdVid 7B [2I
56.4
LLaVA-One Vision 7B 9
56.7
58.2
Qwen2- LL 7B
88.9
86.6
78.2
80.6
82.8
88.8
67.0
63.3 69.0
Qwen2.5-VI.
86.
80.0
73.0
77.3
78.8
88.2
69.6
65.1/71.6
STAVEQ2 7B (Ours)
89.8
87.6
78.7
80.9
83.9
88.9
70.1
66.8/7188
LLa Vision 72B
59.4
66.2/69.55
Lad2 72B [10
62.0
63.4/63.1
LLaVA-Vides 72B (55
70.6/76.9
QW 72B
89.8
87.8
77.9
85.3
84.8
90.4
73.6
/
DL 72B
92.1
88.9
81.9
87.1
89.4
91.8
70.4
73.3 79.1
STAVEQ2 72B (Ours)
92.8
90.1
82.3
87.9
90.3
92.8
74.5
73.9 79.9
GPT-4o
71.9 77.2
solve visual matching problems quite well. This makes visual similarity tasks good way to check whether the model is learning useful video representations.
Based on this idea, create the SSv2-VSM SSv2-TI0 designed to evaluate whether Vidco-LLMs can recognize visual similarity between actions. The dataset contains 8,471 samples, each consisting of two reference videos (with different actions) and third query video. The task is to determine whether the action in the query video matches the first, the second, neither video. See Appendix A for prompt examples and Appendix for further details on the subset's 's composition.
Similarity-Matchinge subset, variation of
We evaluate Qwen2-VI. 2B and 7B, along with their STAVEQQ-fine-tun versions, SSv2-VSM. Because this task emphasizes visual comparison rather than open-ended text generation (like action recognition). it allows isolate the quality of the model's video representations. As shown in Table 3 our STAVEQ2 variants outperform the base models by 2.9% and 3.54%, respectively. This suggests that temporal features can improve performance even in tasks focused visual similarity.
5.5 STAVEQ2 Evaluation Benchmarks
evaluate the performance STAVEQ2 on standard video understanding benchmarks to assess T show that STAVEQ2 consistently outperforms recent state-of-the-art Video-I.LMs in all of the benchmarks, demonstrating the robustness and general applicability of our method.
general video understanding capabilities in temporally challenging scenarios. Results in Table

---

We conducted the VITATECS benchmark results for the Qwen2- VL and Qwen2.5-VV VL family, as they do not provide results for this benchmark themselves. the VITATECS benchmark, we observe that the STAVEQ2 models outperform both their Qwen2-VI counterparts and the newer Qwen2.5-VI. models, achieving the highest scores all aspects, including direction and sequence understanding. On the Video-MMi MME dataset, we outperform other models, including GPT-4o by 2/2.7 and LLa VA- Video 72B by 3.3/3 regarding accuracy. Our STAVEQ2 72B also outperforms other models on MVBench. Notably, STAVEQ2 7B surpasses ST-LLM 7B, model which attempts to delegate the task of modeling spatiotemporal sequences to the LLM, by 15.2 on MVBench. Additionally, STAVEQ2 7B outperforms TG-Vid 7B by 13.7.
6 Conclusions
We introduced STAVEQ2, our Video-LLM architecture with stacked temporal attention modules in the vision encoder, enabling more accurate modeling of temporal relationships and action progression across video frames. Our analysis showed that existing models struggle with temporal reasoning. By enhancing temporal modeling directly at the visual encoding stage, our model improves video question answering performance. These findings suggest that temporal attention at the encoder level crucial for better generalization and temporal understanding in Vidco-LLMs.
Limitations and Future Work Due resource constraints, we did not pretrain the model train from scratch, and our experiments were limited to models up to 72B parameters. Future work can explore full pretraining and scaling beyond 72B to further evaluate the architecture's potential.
Acknowledgments and Disclosure of Funding
This work was partially funded by the Federal Ministry for Transport (BMV), Germany ("MoToRes", 01F2271A) and by the Federal Ministry for Economic Affairs and Energy (BMWE), Germany ("ATTENTION!", 01MJ22012D).
References
[1] Roberto Amoroso, Gengyuan Zhang, Rajat Koner, Lorenzo Baraldi, Rita Cucchiara, and Volker Tresp. Perceive, Query & Reason: Enhancing Vidco QA with Question-Guided Temporal Queries, 2024.
[2] Piyush Bagad, Makarand Tapaswi, and Cees G. M. Snock. Test of Time: Instilling Video-L Ladco-Langgage Models with a Sense of Time, 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang, Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-VE Technical Report, 2025.
[4] Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman, Frozen in time: A joint video and image encoder for and-to-end retrieval. In IEEE International Conference on Computer Vision, 2021.
[5] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is Space-Time Attention All You Need for Vidco Understanding? Proceedings of the International Conference on Machine Learning (ICML), July 2021.
[6] Shimin Chen, Xiaohan Lan, Yitian Yuan, Zequn Jic, and Lin Ma. TimeMarker: A Versatile Vidco-LLM for Long and Short Video Understanding with Superior Temporal Localization Ability, 2024.
[7] Shuo Chen, Zhen Han, Bailan He, Jianzhe Liu, Mark Buckley, Yao Qin, Philip Torr, Volker Tresp, and Jindong Gu. Can Multimodal Large I Language Models Truly Perform Multimodal In-Context Learning? In 2025 IEEF/CVF Winter Conference Applications of Computer Vision (WACV), pages 6000-6010. IEEE, 2025.
[8] Sihan Chen, Handong Li, Qunbo Wang, Zijia Zhao, Mingzhen Sun, Xinxin Zhu, and Jing Liu. VAST: Vision-Audio Subtitle Saati--dddooSSuilee----eee tion Processing Systems, Onni-Modality Foundation Model and Dataset. Advances in Neural Informa- 2023.
[9] Zhe Chen, Weiyun Wang. Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding Performance Boundaries of Open SourCe Multimodal Models with Model, Data, and Test- Time Scaling, arXiv preprint 27577 2024.
10

---

[10] Zesen Cheng, Sicong Leng, Hang Zhang. Yifei Xin, Xin Li. Guanzheng Chen. Yongxin Zhu, Wenqi Zhang. Ziyang Luo, Deli Zhao, and Lidong Bing. VideoLLaMA 2: Advancing Opatial-Temmora Modeling and Audio Understanding Video-LLMMs 2024.
[11] Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang. Shiliang Zhang, Zhijie Yan. Chang Zhou, and Jingren Zhou. Qwen-Audio: Advancing Universal Audio Understanding via Unitied Large-Scale Audio- LanguaLe Models. arXiv preprint arXiv:2311.07911 2023.
[12] Can Cui, Yunsheng Ma, Xu Cao, Wenqian Ye, Yang Zhou, Kaizhao Liang, Jintai Chen, Juanwu Lu, Zichong Yang. Kuci-Da Liao, et al. A Survey Multimodal Large Language Models for Autonomous Driving. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 958-979, 2024.
[13] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In International Conference on Learning Reprexentations, 2021.
[14] Dyke Ferber, Georg Wölffein, Isabella C. Wiest, Marta Ligero, Srividhya Sainath, Narmin Ghaffari Laleh, Omar S. M. Nahhas, Gustav Müller-Franzes, Dirk Jäger, Daniel Truhn, and Jakob Nikolas Kather. In-context learning enables multimodal large language models to classify cancer pathology images, 2024.
[15] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-MME: The first-ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. In Proceedings of the Computer Vision and Pottern Recognition Conference, pages 24118-824111 2025.
[16] Difei Gao, Lei Ji, Luowei Zhou, Kevin Qinghong Lin, Joya Chen, Zihan Fan, and Mike Zheng Shou. AssistGPT: A General Multi-modal Assistant that can Plan, Execute, Inspect, and Learn arXiv preprint a/Xiv:2306.08644) 2023.
[17] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pages 5842-5850, 2017.
[18] Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. MA-LMM: Memory-Augment Large Multimodal Model for Long-Term Video Understanding, 2024.
[19] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, In International Conference on Learning Representations, 2022.
[20] Wenbo Hu, Yifan Xu. Li, Weiyue Li, Zeyuan Chen, and Zhuowen Tu. BLIVA: A Simple Multimodal LLM for Better Handling of Text-Rich Visual Questions. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 2256-2264, 2024.
[21] Zi- Yuan Hu, Yiwu Zhong, Shijia Huang, Michael R Lyu, and Liwei Wang. Enhancing Temporal Modeling of Video LLMs via Time Gating. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 2845-2856, 2024.
[22] Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. VTimeLLM: Empower LLM Grasp Video Moments. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14271-14280, 2024.
[23] Rongjie Huang, Mingze Li, Dongchao Yang. Jiatong Shi, Xuankai Chang, Zhenhui Ye, Yuning Wu, Zhiqing Hong, Jiawei Huang, Jinglin Liu, Yi Ren, Zhou Zhao, and Shinji Watanahe. AudioGPT: Understanding and Generating Speech, Music. Sound, and Talking Head, 2023.
[24] Jindong Jiang, Xiuyu Li, Zhijian Liu, Muyang Li, Guo Chen, Zhiqi Li, De-An Huang, Guilin Liu, Zhiding Yu, Kurt Keutzer, Sungjin Ahn, Jan Kautz, Hongxu Yin, Yao Lu, Song Han, and Wonmin Byeon Token-Efficicnt Efficient Long Video Understanding for Multimodal LLMs, 2025.
[25] Dohwan Ko, Ji Soo Lee, Wooyoung Kang, Byungseok Roh, and Hyunwoo Kim. Large Language Models are Temporal and Causal Reasoners for Video Question Answering. In 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, pages 13000-3311 2023.
11

---

[26] Bo Li. Yuanhan Zhang. Liangyu Chen, Jinghao Wang, Fanyi Pu, Jingkang Yang. Chunyuan Li. and Ziwei
Liu. MIMIC-IT: Multi-Modal In-Context Instruction Tuning. arXiv preprint wrXi::2306.05255, 2023.
[27] KunChang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. VideoChat: Chat-Centric Video Understanding. arXiv preprint arXiv:2305.06335, 2023.
[28] Kunchang Li, Yali Wang. Yinan He. Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. MVBench: Comprehensive Mudti-modal Vidco Understanding Benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1222222,,, 2024.
[29] Shicheng Li, Lei Li, Yi Liu, Shuhuai Ren, Yuanxin Liu, Rundong Gao, Xu Sun, and Lu Hou. VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Models. In Aleš Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol. editors. Computer Vision Cham, 2025. Springer Nature Switzerland. ISBN 978-3-031-72897-6
ECCV 2024, pages
[30] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLa Learning United Visual Representation by Alignment Before Projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 5971-5984, 2024.
[31] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual Instruction Tuning, 2023.
[32] Ruyang Liu, Chen Li, Haoran Tang. Yixiao Ge, Ying Shan, and Ge Li. ST-LLM: Large Language Models Are Effective Temporal Learners. In European Conference on Computer Vision, pages 1-18. Springer. 2024.
[33] Muhammad Maaz, Hamoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-ChalGPT: Towards Detailed Video Understanding via Large Vision and Language Models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12585-12602, 2024.
[34] F Mahdisoltani, G Berger, W Gharbich, D Fleet, and R Memisevic. On the effectiveness of task granularity for transfer leamning. arXiv preprint 1e04vv8804.000 09235, 2018.
[35] Ming Nie, Dan Ding, Chunwei Wang, Yuanfan Guo, Jianhua Han, Hang Xu, and Li Zhang. SlowFocus: Fine-grained Enhancing Fine- grained Temporal Understanding in Video LLM. In A. Globerson. L. Mackey, Belgrave,
A. Fan, U. Paquet, J. Tomezak, and C. Zhang, editors, The Thirty-eighth Annual Conference on Neural Information Processing Systems, volume 37, pages 81808-81823 Curran Associates, Inc., 2024.
[36] Yuxiang Nie, Han Wang, Yanjie Wang, Can Huang, Liang Lin, and Guanbin Li. Video Q-Former: Multi- modal Large Language Model with Temporl Querying Transformer Towards Video Understanding, 2025.
[37] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding Multimodal Large Language Models to the World. arXiv preprint (tt::2306.14822 2023.
[38] Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Li, and Wanxiang Che. What Factors Affect Multi- Modal In-Context Learning? An In-Depth Exploration. Advances in Neural Information Processing Systems, 37: 123207-123236, 2024.
[39] Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Li, and Wanxiang Che. What factors affect multi- modal in-context learnin?? an in-depth exploration. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[40] Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14313-14323, 2024.
[41] Enxin Song. Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, al. MovieChat: From Dense Token Sparse Memory for Long Video Understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18221 -8832, 2024.
[42] Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Cc Liu, and Lijuan Wang. GIT: A Generative Image-to-text Transformer for Vision and Language. arXiv preprint arXiv:2205. 14100, 2022.
[43] Limin Wang, Bingkun Huang. Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. VideoMAE V2: Scaling Video Masked Autoencoders with Dual Masking, 2023.
12

---

[44] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu. Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-VL: VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution, 2024.
[45] Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Lu Yuan, and Yu-Gang Jiang. Masked Video Distillation: Rethinking Masked Feature Modeling for Self-supervised Video Representation Learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6312-6322, 2023.
[46] Yi Wang. Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang. al. Video: General Video Foundation Models via Generative and Discriminative Learning. arXiv preprint arXiv:2212.031191, 2022.
[47] Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan Hc, Guo Chen, Baoqi Pei, Rongkun Zheng, Zun Wang, Yansong Shi, al. Internvide22: Scaling Foundation Models for Multimodal Video Understanding. In European Conference on Computer Vision, pages 396-416. Springer, 2024.
[48] Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang. Changlian Ma, Haian Huang, Jianfei Gao, Min Dou, Kai Chen, Wenhai Wang, Qiao, Yali Wang, and Limin Wang. Vidco2::: Empowering Vidco MLLMs with Long and Rich Context Modeling, arXiv preprint arXiv:2501. 12386, 2025.
[49] Zhanyu Wang, Longyue Wang, Zhen Zhao, Minghao Wu, Chenyang Lyu, Huayang Li, Deng Cai, Luping Zhou, Shuming Shi, and Zhaopeng Tu. GPT4Video: A Unified Multimodal I Large I Language Model for ciaruction-Followeer Understanding and Safety-Aware Generation. In Proceedings of the 32nd ACM International Conference on Multimedia, pages 3907-3916, 2024.
[50] Xilin Wei, Xiaoran Liu, Yuhang Zang. Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, al. VideoRoPE: What Makes for Good Video Rotary Position Embedding? In International Conference on Machine Learning, 2025.
[51] An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang. Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang. Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng Hc, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang. Mci Li, Mingfeng Xuc, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 Technical Report, 2024.
[52] Chaoyi Zhang, Kevin Lin, Zhengyuan Yang. Jianfeng Wang, Linjie Li, Shung-Ching Lin, Zicheng Liu, and Lijuan Wang. MM-Narrator: Narrating Long-form Videos with Multimodal In- In-Context Learning, 2023.
[53] Hang Zhang, Xin Li, and Lidong Bing. Video-I LaMA: An Instruction-tuned Audio- Visual Language Model for Video Understanding. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 543-553, 2023.
[54] Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. LLaVA-NeXT: A Strong Zero-shot Video Understanding Model, April 2024.
[55] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video Instruction Tuning With Synthetic Data, 2024.
13

---

A Prompt Examples
This section provides example prompts used for our experiments.
Box 1: Example prompt with in-context examples for SSv2-T10 dataset
Instruction:
Look at the provided examples and answer the last question.
Example <video> The action happening in this video is: Moving [something] from left to right.
Example <video> The action happening in this video is:
Moving [something] from right left.
Final Prompt- <video> Now considering the previous examples, what action is happening in this video?
Box Example prompt for SSv2-VSM dataset
Instruction: Look at the provided examples and identify which example related to the final video.
Example <video> The action happening in this video is:
Moving [something] away from [something].
Example 2 <video> The action happening in this video is:
Moving [something] closer to [something].
Final Prompt- <video> Now considering the previous examples, is there any action related to this video? If not, respond with "No related action" and there is, respond with the example number and action.
Box Example Prompt for Evaluation
Instruction: Look at the ground truth and the LLM's answer. Decide whether the LLM's s answer matches the ground truth.
Ground Truth: Pulling [something] from left to right
LLM Answer: The action is moving something from right to left on the floor
Question: Based on the ground truth, is the LLM answer correct? Answer with
simple "Yes" or "No".
Box [ illlsttraees the prompt structure used in our few-shot experiments on SSv2-T10 dataset. In this setup, the model is presented with one more in-context examples, each consisting of video and its corresponding action label. After reviewing these examples, the model is tasked with inferring the action occurring in new query video. This format encourages the model to generalize from the provided samples and demonstrate its capability to recognize similar actions.
Box presents the prompt format used for the SSv2-VSH -SM dataset, which is designed to evaluate the model's 's ability to perform similarity matching. The model provided with multiple labeled video examples and a final query video, and it must determine whether any of the examples depict an action related the one shown
query. a related action exists, the model is expected return example number along with the action label; otherwise, it should respond with "No related action." This task format emphasizes fine-grained action discrimination and serves as a valuable test of the model's 's capacity for visual-semanti matching.
14

---

Table Action categories and their relative frequencies in SSv2-T10.
Action
Frequency (%)
Pulling [something] from left right
14.68
Pulling [something] from right to left [Something] falling like a rock
14.97
13.12
Picking [something] up
9.26
Throwing [something] the air and letting it fall
7.31
Throwing [something] in the air and catching it Moving [something] away from [something]
8.90
8.61
Moving [something] closer to [something]
8.55
Rolling [something] on a flat surface
11.85
Poking stack of [something] the stack collapses
2.74
Table 6: Ablation study results order of spatial and temporal attention and head scaling in STAVEQ2 2B on SSv2-TIO. Accuracy (%) is reported, with the best result bold.
Model
Attention Order
Head Scale
Acc (%)
Qwen2- VI. 2B
1.0
73.14
STAVEQ2 2B
Spatial First
1.0
58.34
STAVEQ2 2B
Spatial First
0.5
71.18
STAVEQ2 2B
Temporal First
0.25
73.20
STAVEQ2 2B
Spatial First
0.25
76.04
In open-ended generation, the model's output may not precisely match the expected action name. To address this, we employ another LLM. Qwen2-7B, judge evaluate the responses generated by Video-LLMs. Box provides the evaluation prompt. Given ground truth label and the model's predicted answer, the judge determines whether the prediction correct, returning a binary "Yes" or "No" response.
B SSv2-T10 Composition
Table [5] lists the action categories in the SSv2-T10 split and their relative frequencies (percentage of examples). These ten actions were used both to select representative in-context examples and to design the prompts used our experiments.
C Ablation Studies on STA Architecture
To investigate the design of our stacked temporal attention, we conduct ablation studies STAVEQ2 2B, fine-tuned on the SSv2-T10 dataset. The first study examines the internal configuration of STA-enhanced transformer blocks, including the order of spatial and temporal attentions (i.e. whether temporal attention should placed either before or after spatial attention) and the number of temporal attention heads. The second study investigates the placement of STA-enhanced transformer blocks across the vision encoder.
As summarized in Table 6 positioning temporal attention after spatial attention with a head scaling factor of 0.25 achieves the highest accuracy (76.04%), outperforming the baseline Qwen2-VL 2B (73.14%) by 2.90%. Reducing the number of temporal attention heads (e.g., 0.25 vs. 1.0 relative to the baseline number of heads) enhances performance, likely due to improved regularization and focus on critical temporal features, particularly when there is limited data, aligning with our emphasis on parameter efficiency. Placing temporal attention before spatial attention (73.20% at 0.25 scale) yields slightly lower performance, indicating that processing spatial context first enhances temporal modeling. These results validate the design our stacked temporal attention, especially for fine-graines temporal understanding tasks.
15

---

Table Ablation study results on number and placement of STA-enhanced transformer blocks in STAVEQ2 2B on SSv2-T10. Accuracy (%) is reported, with the best result in bold.
Model
# Temporal Blocks
Placement
Acc (%)
Qwen2-V VL 2B
73.14
STAVEQ2 2B
16
Uniform
74.73
STAVEQ2 2B
16
First blocks
74.97
STAVEQ2 2B
32
All blocks
76.04
Table 8: Performance of Intern Video2-Chat, fine-tuned SSv2-T10. Adding stacked temporal
attention (STA) leads to a significant accuracy gain.
Method
Acc (%)
Intern Video2-Chat 8B
84.17
Intern Video2-Chat 8B STA
95.18 (f 11.01%)
Building on the optimal internal configuration from Table we assess the impact the number and placement of STA-enhanced transformer blocks across the vision encoder in STAVEQ2. According to the results summarized in Table 7 using 32 STA-enhanced transformer blocks across all layers achieves the highest accuracy (76.04%) outperforming the baseline Qwen2-VL. 2B (73.14%) by 2.90%. Reducing the number of STA-enhanced blocks to 16, whether distributed uniformly (74.73%) or concentrated in early layers (74.97%), results in a lower performance. The minimal difference between uniform and early-layer placement indicates that strategic block positioning has less impact than the total number of blocks.
D STA Enhancement Effect on Viterr-iieeo2-hat
To complement the qualitative analysis in Section 3.3 (Figures
Table provides the corre- sponding quantitative performance metrics for Intern Video2-Chat 8B on SSv2-T10, both before and after applying stacked temporal attention (STA). Note that Intern Video2-Chat is limited to processing up to 8 input frames, making it suitable only for short-video benchmarks like SSv2-T10 and excluding from evaluations on longer-video datasets such as those in Table 4 The baseline InternVideo2. Video2 -Chat 8B model achieves an accuracy of 84.17%. With the integration of STA, the performance improves significantly, reaching 95.18%-4. substantial gain of 11.01% These results underscore the effec- tiveness of STA in boosting the temporal understanding capabilities of Video-LLMs, particularly for fine-grained action recognition tasks.
E SSv2-VSM Dataset Composition
As described in the paper, each sample in SSv2-VSM dataset consists of two reference videos (with different actions) and a third query video. The task is to determine whether the action in the query video matches the first, the second, neither video. For the SSv2-VSM dataset, explored the optimal ratio of positive to negative samples for fine-tuning the models. Positive samples consist of reference videos where one matches the query video's action, while negative samples have no matching actions with the query video.
Table shows that increasing the proportion of positive samples generally enhances performance, with similarity matching accuracy improving from 25.52% at 50% positive samples to a peak of 71.25% at 80% positive samples. However, accuracy drops to 49.18% at 91% positive samples, indicating that an excessively high proportion of positive samples may reduce dataset diversity and hinder generalization.
In our experiments reported in Table 9 we include textual descriptions of the actions happening in each reference video. However, removing these textual descriptions increases task difficulty, as the
16

---

Table Context-Selection tin-ttning results with varying positive-negation sample ratios. Similarity Matching evaluates whether the model can correctly identify relevant context sample among distractors when such a sample present.
Dataset Composition
Accuracy (%)
50% positive
25.52
80% positive
71.25
91 % positive
49.18
similarity matching accuracy at 80% positive samples drops from 71.25% to 68 68.65% by removing the descriptions. Consequently, to evaluate the models on more challenging task, excluded textual descriptions for reference videos in the main experiments reported in the paper.
F Prompting Experiments
in in-context learning performance in image and text modalities, demonstrating that instruction placement and modality ordering can impact performance. To understand if these findings generalize to the video domain, conducted a parallel set of controlled experiments on the SSv2-VSM dataset.
Recent work [39] systematically investigated the factors affecting
First, we evaluated the impact of instruction placement by testing four prompting strategies, adapting the methodology from [39]] No-Instruction, where the prompt contains no explicit task instruc- tion; Introductive, where task description appears once at the start of the prompt, before the context examples; Introductive-Summiin
where an introductory instruction is used, along with a summary instruction that appears after the context examples but before the final query; and Intra- demonstration, where the task instruction is repeated within each demonstration example. As shown in Table [O] the choice of instruction style had minimal effect on the performance of most models. However, we observed a notable exception: Intern Video2-Chat performance improved significantly (by 9%) with the Serrodduciiv-SSmmattt strategy. Sample prompts illustrating each strategy provided in Box
Second, we tested the effect of intra- intra-demoossratioon modality ordering that is, whether the sequence of the video and corresponding text tag within each context sample affects model understanding. We compared two formats:
(where text precedes the <video> tag) and Video-Texs (where the <video> tag appears first). The results in Table 11 again show a strong model-dependent sensitivity. While the ordering had minimal impact the performance of most models, it critical factor for Intern Video2-Chat This model performed significantly better (37.06% vs. 15.98%) when the video presented first (Video-Texe Text), aligning with findings for image- image-based models in
Box 4: No-Instruction example prompt for SSv2-VSM
Instruction:
Example <video> The action happening this video is: Throwing [something] in the air and catching it.
Example 2 <video> The action happening in this video is: Pulling [something] from left right.
Final Prompt- <video> now considering the previous examples, is there any action related to this video? If not, respond with No related action and if there is, respond with the example number and action.
17

---

Box 5: Introductive example prompt for SSv2-VSM
Instruction: Look the provided videos and identify which video is related to the final video.
Example <video> The action happening this video is: Throwing [something] in the air and catching it.
Example 2 <video>> The action happening in this video is: Pulling [something] from left right.
Final Prompt- <video> now considering the previous examples, is there any action related to this video? not, respond with No related action and if there is, respond with the example number and action.
Box 6: Introductive- Sarroducivv-Summative example prompt for SSv2-VSM
Instruction: Look at the provided videos and identify which video related to the
final video.
Example <video> The action happening in this video is: Throwing [something] in the air and catching
Example 2 <video> The action happening in this video is: Pulling [something] from left to right.
Summary: In summary, the two given videos each contain an action taking place, which has been provided to you. You need to recognize these actions and keep them in mind for the next question.
Final Prompt <video> now considering the previous examples, there any action related to this video? If not, respond with No related action on and if there is, respond with the example number and action.
Box 7: Intra-demonstrrttoo example prompt for SSv2-VSM
Instruction:
Example 1 <video>> The action happening in this video is: Throwing [something] in the air and catching it. So this is the video number 1, remember the video and the action taking place, you need to see if another video has a similar action later.
<video> The action happening this video Pulling [something] from left to right. So this is the video number 2, remember the video and the action taking place, you need to see if another video has a similar action later.
Example 2
Final Prompt <video> now considering the previous examples, there any action related to this video? If not, respond with No related action and if there is, respond with the example number and action.
G Attention Visualization
To qualitatively evaluate the impact of stacked temporal attention (STA), we visualized the attention maps generated by InterVVideo2 IB vision only model before and after applying STA. Figure A shows person poking a lighter so that it falls, labeled with the class poking [something] so that it falls over. Before applying STA, the attention maps show that the model places minimal focus on the lighter, despite the fact that the action class is primarily defined by the lighter's movement and fall. After
18

---

Table 10: Prompt structure impact on SSv2-VSM performance. We compare four prompting styles: No-Instructt co-tunsruction, Introductive (task instruction at the beginning), Introductive-Summant Summative (instructions both at the beginning and after context), and Intra-demonstration (instruction repeated before each context sample).
Introductive- Summative
Intra-
Model
No-Instruction Introductive
demonstration
LLaVA-NeXT-Video 7B
20.08%
20.11%
20.11
20.11%
Owen2-VL. 2B
22.76%
21.29%
21.46%
24.97%
Qwen2-VL 7B
20. 14%
20.11%
20.22%
20.28%
InternVideo--Chat 8B
35.94%
37.06%
45.02%
34.94%
Table 11: Effect of intra-context tag ordering on ESv2-VSM performance. We compare two context formats: Text- Video, where the text appears before the video tag in each context item, and Video-Texx where the video tag comes first.
Model
Text-Video
Video-Text
VLaVA-NeTT-Video 7B
20.11%
20.11%
Qwen2-VL
24.09%
21.29%
Qwen2-Vi 7B
22.32%
20.11%
Intern Video2-Cht 8B
15.98%
37.06%
incorporating STA, the model's attention shifts significantly toward the lighter, especially as it begins fall, indicating improved temporal modeling and relevance attribution.
These visualizations underscore the effectiveness of stacked temporal attention in helping the model focus temporally relevant regions of the video. Especially in cases where the action depends subtle object movements over time, STA enhances the model's ability to localize and interpret key interactions, thereby improving classification performance.
H
Extra Models
While primary contributions center enhancing the Qwen2-V2 LL architecture with stacked temporal attention STAVEQ2, we further validate the broad applicability this mechanism by extending it several very recent and competitive Video-LLM architectures. These extensions demonstrate that STA is not tied to specific base model but provides consistent improvements in temporal understanding across diverse design paradigms- -iniisdinn those that already incorporate advanced temporal modeling techniques. Due to resource constraints, we focused on representative recent models that align closely with our evaluation benchmarks.
For the newer Qwen2.5-VI. we introduce STAVEQ2.5, which integrates dedicated temporal attention blocks into its vision encoder. Qwen2.5-VL. employs windowed attention in most layers and full spatial self-attention in only four layers; however, our approach remains unaffected, as cach patch in the STA block attends to corresponding patches across all frames, bypassing the limitations imposed by windowed attention. This design preserves the temporal attention mechanism of STAVEQ2. ensuring consistent temporal modeling. We train STAVEQ2.5 using the same two-stage strategy employed for STAVEQ2, leveraging the WebVid-QA dataset.
We also apply STA to VideoRoPE 7B [50] state-of tort-e--rtttttttt model based on Qwen2-Vi VL that enhances temporal awareness through improved positional embeddings for output feature tokens fed to L.M.. In contrast, our STA targets token-level temporal interactions directly within the vision encoder, making the two approaches complementary rather than competing. Our experiments confirm this synergy: integrating STA with VidcoRoPE yields further gains, highlighting how dedicated temporal
19

---

(a) Before STA
(b) After STA
Figure Attention maps for the action poking [something] so that it falls over.
Table 12: Accuracy (%) on video understanding benchmarks for our STAVEQ2.5 compared to other models. For VITATECS, aspect-wise results are shown; other benchmarks report overall accuracy. IV2.5-Chat refers Intern IntemViideo2.5Chail *(Video-MME without/with subtitles). Results collected from the Video-MME leaderboard. indicates results not reported in the original paper and unavailable from other sources.
VITATECS
Model
MVBench * *VMME (wo/w)
Comp.
Dir.
Int.
Loc. Seq.
Type
Qwen2-VL 2B
80.8
82.1
69.6
76.1
72.2
85.9
63.2
55.6 60.4 60.4 56.2 61.3
STAVEQ2 2B (Ours)
81.3
83.0
70.1
76.9
72.9 86.6
65.1
ST-LLM 7B
54.9
TG-Vid 7B
56.4
LLaVA-OneViision 7B
56.7
58.2
VideoRoPE
81.1
81.8
60.9
79.4
80.7
85.8
57.3
61.6/
VideoRoPE + STA (Ours)
81.9
82.9
61.8
79.9
81.3
86.3
59.2
62.5
Qwen2- VL 7B
88.9
86.6
78.2
80.6
82.8
88.8
67.0
63.3/69.0
Qwen2.5- 7B
86.1
80.0
73.0
77.3
78.8
88.2
69.6
65.1/71.6
STAVEQ2 7B (Ours)
89.8
87.6
78.7
80.9
83.9
88.9
70.1
66.8/7188
STAVEQ2.5 7B (Ours)
88.0
82.1
74.2
77.9
79.7
88.9
70.3
66.2/72.5
Intern Video2.55-hhal 8B
91.3
88.7
82.0
84.8
84.7
91.0
75.7
65.1
IV2.5-Chat 8B + STA (Ours)
91.6
89.7
82.7
85.6
85.8
91.3
76.8
65.9
LLaVA-One Vision 72B
59.4
66.2
Lad2 72B
62.0
63.4/63.1
ALaVA-Vided 72B
70.6
Qwen2- 72B
89.8
87.8
77.9
85.3
84.8
90.4
73.6
71.2 7
Qwen2.5-Vis 72B
92.1
88.9
81.9
87.1
89.4
91.8
70.4
73.3 79.1
STAVEQ2 72B (Ours)
92.8
90.1
82.3
87.9
90.3
92.8
74.5
73.9
79.9
STAVEQ2.5 72B (Ours)
93.1
90.9
82.1
88.0
90.8
93.3
72.4
74.2
79.8
GPT-4o
71.9 77.2
attention can amplify existing enhancements. Finally, we extend STA to Intern Video2.5-Chat 8B 48 recent evolution of the Vntern family that supports longer video inputs (unlike Intern Video2- Chat that limited to processing 8 input frames). Intern Video2.5-Chaa is trained using the same approach as other models and evaluated on the full of benchmarks.
Evaluation on video understanding benchmarks- -VIAAA.SSSSIISSII...S MVBench, and Video-MME demonstrates the effectivvnnes of these extensions in handling temporally complex scenarios. The results show that our STA approach not only enhances Qwen2-Vi VL but also improves these newer architectures, confirming its wide applicability.
As shown Table 12 STAVEQ2.5 consistently outperforms its base model, Qwen2.5- VL,, across VITATECS, MVBench, and Video-MME, validating the effectiveness of our stacked temporal attention approach enhancing video understanding. Notably, STAVEQ2.5 72B surpasses STAVEQ2 72B on most benchmarks, demonstrating superior performance of our approach when applied to the
20

---

newer Qwen2.5-Ve VL architecture. Similarly, applying STA to VideoRoPE and Intern Video2.5-Chat yields consistent gains, further highlighting the method's robustness and its ability synergize with orthogonal temporal enhancements. These results underscore the adaptability of our stacked temporal attention mechanism, which drives performance improvements across Video-LLMS of varying scales and architectures, excelling in tasks that demand sophisticated temporal understanding.
21

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes].
Justification: We have stated the contributions and scope of the paper in the abstract and introduction of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes].
Justification: We discuss the limitations of the work caused by limited computation resources in Section [ as well as the future works that we have not done and are potential ideas for future research papers.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "I "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
22

---

Answer: [NA].
Justification: Our paper does not focus on the theoretical aspect. We provide extensive evaluation our method.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes].
Justification: We have explained our method in Section Method. Since have used open- source models and datasets, everyone can easily reproduce our experiments and results. We have also provided the supplementary materials, including our code.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
23

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
The code of our paper is provided as supplementary material and will be made available in GitHub repository upon paper publication.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://niis.cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https for more details.
7 //nips. //nips.cc/pub/lle/ wwtpc//pppppg..deeeeeeeepeeeppee httpl///guidee//ddeuubiiiiseee irnooeSuumissionPol111.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All of the experiment setups are included with our code as supplementary material. Also, the most important and general setups of our training and test are mentioned in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Due to the large size of video datasets and the high computational cost, repeating experiments report statistical significance is generally not feasible in this field.
Recent top-tier Video-LLM works like Video-LLaVA 30 and Qwen2-VI.fa [4 also do not report such significance tests for similar reasons.
Guidelines:
The answer NA means that the paper does not include experiments.

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have mentioned the experiments setups and compute resources in most important parts of the experiments, for example Section But according to the number of experiments and the page limit of the template we couldn't 't report them for all of the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https::///// //teurii///rrrps.... http:pps....ppppiceicccccccatdda
Answer: [Yes]
Justification: Yes, the research conform with the NeurIPS code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA].
25

---

Justification:
Our paper is a foundational research and there is no societal impact of this work.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We can't 't assume any misuse of our model which can be used for video-question answering and we have used open- open-source models and dataset that this criteria has been already considered in them.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging. and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes].
Justification: Yes, they are cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
26

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA].
Justification; We have released any new assets datasets. However, we provide different derivates of an existing dataset SSomething-Somethnng SSmeehing V2) that we use for training and evaluation. We make these available in supplementary material, using IDs to refer to the videos in the original dataset.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA].
Justification: We have not done any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
27

---

Justification:
We have not done any crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality the research, declaration is not required.
Answer: [Yes].
The paper's topic is Large Multimodal Language Models with the focus on videos. So LLMs are a core component our studied architecture and we have described the usage of LLMs for tasks such as video question answering.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy autpee//heurpp.cccooofrrrrreee/e for what should or should not be described.
28


Reviews:
[{"review_id": "3kBCw6UX2A", "reviewer": "Reviewer_58rP", "summary": "The paper proposes Stacked Temporal Attention to be a mechanism to improve the temporal awareness of the vision encoder used in VLMs. Specifically, it applies the attention across tokens on the same spatial location across times, in addition to the per-frame attention. It created a subset of SSv2 and show that the proposed modification improve the performance on this specific set covering action classes that particularly require temporal awareness. Also, the authors benchmark the proposed approach on standard video understanding benchmarks like MVBench, VideoMME etc.", "questions": "- For the fine-tuning results in Figure 2c, what's the size of the held-out set on SSv2-T10? \n- Typos: L201, S^{(m)}1, i -> S^{(m)}_{1, i}\n- Typos: Eq 7, \"Z^{(m)}\" -> \"S^{(m)}\", according to the texts in L206", "limitations": "Yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 1, "strengths_and_weaknesses": "- Table 1, the low performance with visual prompting is not surprising, as most VLM models are not trained with enough number of multi-turn visual prompting data. Thus IMO this visual prompting experiment does not add much information upon the zero-shot results. \n- The proposed modification could in principle be applied to other vision-encoder/VLM, beyond Qwen2 models. It would be more useful if the authors could show some results generalizing it beyond Qwen2 models. \n- There is limited novelty in this paper, as the proposed \"Stacked Temporal Attention\" has been explored before (e.g. in [1]) and studied as one variant of the space-time transformers. \n- There is another line of work aiming to improve the temporal awareness via better encoding (e.g. VideoRoPE [2]), which should be included as the baseline for this work. \n\n[1] Is Space-Time Attention All You Need for Video Understanding?, Gedas Bertasius, Heng Wang, Lorenzo Torresani\n[2] VideoRoPE: What Makes for Good Video Rotary Position Embedding?, Wei et al. ICML 2025", "was_revised": true, "final_justification": "I thank the reviewers for their efforts in the rebuttal. I appreciate the added baselines and discussions on VideoRoPE and InternVideo2.5-Chat. To reiterate, I never argued against the importance of incorporating temporal attention in vision encoders for LLM, what I concerned about is the fact that there has been prior work studying the interplay of spatial and temporal attention (albeit for video classification task) and I think this paper has not addressed that fact enough. That said, I do agree that the current trend of designing *vision encoder for LLM* is not putting enough focus to incorporate temporal attention and this paper could serve to raise attention on that. Given that, I will raise my rating to borderline accept the paper into the conference.", "rebuttal": "We thank you for the detailed comments. Below, we address each concern and provide clarifications, results, and plans for the camera-ready version.\n\n**W1:** Visual prompting experiments\n\nOur main goal here is to properly analyze current state-of-the-art foundational models for their basic temporal understanding capabilities. Usually when some capabilities are missing in a family of models one of the important baselines and ablations is to try to simply train the models to achieve that capability. Apart from convectional training approaches, another way to teach a LLM a new capability/skill is to use few shots samples and in context learning. Some recent studies [1,2] show that multimodal LLMs benefit from in-context learning without further training needed. To this end, we also included a visual prompting ablation study to properly analyze the temporal understanding capabilities of current state-of-the-art foundational video LLMs. Our experiment illustrates a critical point: *even with proper visual prompting, current video LLMs fail to capture fine-grained temporal dependencies*, which are essential for accurate video understanding. We conduct all these experiments to properly identify the current capabilities of state-of-the-art foundational video LLMs and identify the problem, *as mentioned by all of the other reviewers as our strengths:*\n\n*\"The problem is well-identified\" (**reviewer q8eF**)*\n\n*\"Great analysis of the current limitation on visual understanding of LVLM.\" and \"Following problem identification and their approach also sounds reasonable.\" (**reviewer rKct**)*\n\n*\"The authors conduct a thorough analysis of the limitations of current Video-LLMs, providing valuable insights into their shortcomings in temporal reasoning. This analysis motivates the proposed approach and underscores the importance of temporal modeling in video understanding tasks.\" (**reviewer tYNw**)*\n\n\n**W2:** Qwen2-VL Models\n\nThanks for raising this point. We want to clarify that our approach is not limited to Qwen2-VL. In the submitted version, we evaluated **STAVEQ2** (Subsection 5.5, Table 4), which builds upon **Qwen2-VL**, and **STAVEQ2.5**  (Appendix F, Table 6), which builds upon **Qwen2.5-VL**. For this rebuttal, we further applied our method to other recent and highly competitive Video-LLMs, **VideoRoPE [3]** and **InternVideo2.5-Chat [4]**, which also aim to improve temporal awareness. We observed consistent improvements across all these multimodal Video-LLMs. Additionally, we demonstrated that applying Stacked Temporal Attention to the **InternVideo2** vision foundation model achieves a new **state-of-the-art on the SSv2** dataset. This broad evaluation demonstrates strong generalization across different model families and design paradigms, validating that our proposed mechanism is widely applicable. We will include these additional results in the camera-ready version.\n\nBroader evaluation of all models would be ideal but is infeasible due to limited GPU resources.\n\nHere are the results for the newly experimented models:\n\n\n**Table 1**\n\n| Model                     | VITATECS                |           |           |           |           |           | MVBench | VideoMME(wo) |\n|:-------------------------:|:-----------------------:|:---------:|:---------:|:---------:|:---------:|:---------:|:-------:|:------------:|\n|                           | Comp.                   | Dir       | Int.      | Loc.      | Seq.      | Type      |         |              |\n| VideoRoPE[3]                 | 81.1                    | 81.8      | 60.9      | 79.4      | 80.7      | 85.8      | 57.3    | 61.7         |\n| VideoRoPE+STAVE           | **81.9**                    | **82.9**      | **61.8**      | **79.9**      | **81.3**      | **86.3**      | **59.2**    | **62.5**         |\n| InternVideo2.5-Chat[4]       | 91.3                    | 88.7      | 82        | 84.8      | 84.7      | 91        | 75.7    | 65.1         |\n| InternVideo2.5-Chat+STAVE | **91.6**                    | **89.7**      | **82.7**      | **85.6**      | **85.8**      | **91.3**      | **76.8**    | **65.9**         |\n\n\n\n**W3:** Novelty\n\nThe mentioned prior work [5] primarily focuses on end-to-end video classification models. However, in this work we focus on the possibility of improving temporal understanding capabilities of the pretrained vision encoder models that output vision tokens to a generative LLM. Architecturally, our Stacked Temporal Attention (STA) blocks are designed to be additive and parameter-efficient. This preserves the powerful pre-trained spatial weights of the vision encoder and ensures training stability, making it a more viable approach than training a model from scratch or performing a destructive architectural change. We *systematically demonstrate the necessity of having temporal attentions in Video-LLMs*, where it has been largely overlooked or dismissed by recent architectural trends. For instance, Qwen2-VL has only focused on developing Multimodal Rotary Position Embedding for temporal understanding, while relying solely on spatial attention without explicit temporal modeling in the vision encoder, as stated in their paper [6].\n\nOur comprehensive results, presented in Subsection 5.5, Table 4, and Appendix Section F, Table 6 of our paper, and further supported by Table 1 in our rebuttal, directly *challenge this design paradigm*. These results unequivocally show that incorporating explicit temporal attention mechanisms, like our Stacked Temporal Attention, leads to substantial performance improvements on benchmarks that specifically require robust temporal understanding. We will emphasize this positioning more clearly in the camera-ready version.\n\nAnother compelling proof of this is our experiment on the InternVideo2 vision encoder—the previous state-of-the-art (SOTA) model on SSv2, which already incorporates spatiotemporal attention. Our InternVideo2+STA model surpassed this powerful baseline and achieved a new SOTA on SSv2 classification (Subsection 5.2, Table 2). This critical result shows that dedicated, explicit temporal modeling enhances performance even on top of an existing spatiotemporal mechanism.\n\nThis is also reflected in the positive feedback from other reviewers: **Reviewer q8eF** notably described our method as *\"conceptually simple, yet effective,\"* calling it a *\"clear architectural improvement,\"* while **reviewer tYNw** described our work as *\"a novel approach that effectively addresses the limitations of current Video-LLMs in temporal understanding.\"*\n\n\n**W4:** VideoRoPE\n\nAlthough there are other lines of research that aim to improve the temporal understanding of Video-LLMs, such as VideoRoPE [3], they are orthogonal to our approach. To better study this, we have conducted additional experiments incorporating VideoRoPE as a baseline. \n\nIt's important to clarify the distinct focuses of our approaches: VideoRoPE primarily addresses the positional embedding of output feature tokens fed to the LLM, whereas our Stacked Temporal Attention (STA) focuses on enhancing token-level temporal interactions directly within the vision encoder. These are complementary improvements, and our new experiments confirm this: applying STA to VideoRoPE consistently improves its performance across various benchmarks (see Table 1 of our rebuttal). We will include these results and highlight this synergistic relationship in the camera-ready version of the paper.\n\n\n\n**Q1:** SSv2-T10 info\n\nFor evaluating both the **base** and **STA-added models** on SSv2-T10, we used a subset of **2,116** samples with this distribution:\n\n - Pulling [something] from left to right – 14.68%\n\n- Pulling [something] from right to left – 14.97%\n\n- [Something] falling like a rock – 13.12%\n\n- Picking [something] up – 9.26%\n\n- Throwing [something] in the air and letting it fall – 7.31%\n\n- Throwing [something] in the air and catching it – 8.90%\n\n- Moving [something] away from [something] – 8.61%\n\n- Moving [something] closer to [something] – 8.55%\n\n- Rolling [something] on a flat surface – 11.85%\n\n- Poking a stack of [something] so the stack collapses – 2.74%\n\nWe will include the exact details in the appendix/supplementary document of our camera ready version.\n\n**Typos**\n\nThanks for pointing out **the typo in line 201**. We will fix this for the camera-ready version. \n\nHowever, there is **no typo in Equation 7**, and it is indeed correct. As shown below, our architecture incorporates residual connections within each attention block, taking the input to that block and adding it to its output. We will ensure the accompanying text in the camera-ready version clearly reflects this. We appreciate you highlighting these details.\n\nHere are the equations for clarity:\n\nSpatial attention block:\n\n$$\nS^{(m)} = \\text{(concatenation of\\ } S^{(m)}_t \\text{\\ vectors along the temporal dimension)} + \\text{(the residual connection from\\ } X^{(m-1)})\n$$\n\nTemporal attention block:\n\n$$\nZ^{(m)} = \\text{(concatenation of\\ }  Z^{(m)}_i  \\text{\\ vectors along the spatial dimension)} +  \\text{(the residual connection from\\ }  S^{(m)})\n$$\n\nFinal block:\n\n$$\nX^{(m)} = MLP(LN(Z^{(m)})) + Z^{(m)}\n$$\n\n**References**\n\n[1] In-context learning enables multimodal large language models to classify cancer pathology images, Dyke Ferber et. al., Nature 2024.\n\n[2] MM-Narrator: Narrating Long-form Videos with Multimodal In-Context Learning, Chaoyi Zhang et. al., CVPR 2024.\n\n[3] VideoRoPE: What Makes for Good Video Rotary Position Embedding?, Xilin Wei et. al., ICML 2025.\n\n[4] InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling, Yi Wang et. al. arXiv preprint arXiv:2501.12386 2025.\n\n[5] Is Space-Time Attention All You Need for Video Understanding?, Gedas Bertasius et. al., ICML 2021.\n\n[6] Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution, Peng Wang et. al. arXiv preprint arXiv:2409.12191 2024."}, {"review_id": "zs3pAJPYoL", "reviewer": "Reviewer_q8eF", "summary": "This paper presents STAVEQ2, a novel video language model (Video-LLM) architecture that explicitly incorporates stacked temporal attention modules within the vision encoder. The motivation is that existing Video-LLMs struggle with fine-grained temporal reasoning, particularly in action recognition tasks requiring understanding of motion direction or temporal order. The authors demonstrate limitations of existing Video-LLMs (e.g., Qwen2-VL, InternVideo2) on temporally challenging benchmarks. They propose STAVEQ2, which adds efficient temporal attention after spatial attention blocks in the vision encoder. Showing performance gains over strong baselines on multiple video understanding benchmarks, including SSv2, MVBench, VITATECS, and Video-MME.The authors will release code (as a supplement) and propose both action recognition and visual similarity tasks to evaluate temporal modeling.", "questions": "Could the authors provide qualitative visualizations of what the temporal attention captures (e.g., token-level attention across frames)? This would strengthen interpretability.\n\nHow would STAVEQ2 scale to long-form video (e.g., >30 seconds)? Are the temporal attention blocks placed uniformly across layers, or could they be adaptively placed?\n\nHow does STAVEQ2 compare to alternative temporal modeling strategies such as divided space-time attention or hierarchical token merging? Are there trade-offs?\n\nCan the same architecture generalize well to tasks like video captioning or moment retrieval? Would different tuning be needed?\n\nCould the authors isolate the effect of using 1D RoPE vs. 2D or the inclusion of LoRA adapters during training?", "limitations": "The paper briefly discusses limitations related to computational constraints and model scaling (Section 6), which is fair. However, it would be helpful to examine whether temporal modeling introduces brittleness to frame sampling or frame rate changes. Any risks of overfitting to short clip-based benchmarks like SSv2.Potential societal impacts of enhanced video understanding (e.g., surveillance, deepfake detection), if relevant.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe problem is well-identified — existing Video-LLMs poorly capture temporal dynamics.\n\nIncorporating stacked temporal attention directly into the vision encoder is a clear architectural improvement, conceptually simple yet effective.\n\nThe paper is evaluated on multiple benchmarks, including both established (SSv2, MVBench) and diagnostic (VITATECS) benchmarks, showing consistent gains.\n\nAblation and component study: Shows substantial gains over baselines like InternVideo2 with fewer parameters, especially on temporally mirrored action classes.\n\nThe design is lightweight, using fewer attention heads and RoPE adaptations, making it appealing for real-world use, with Table 4 showing performance across a range of model sizes\n\nSupplementary material includes code and implementation details.\n\nWeaknesses:\n\nWhile effective, the idea of adding temporal attention blocks is a relatively incremental change and follows known design principles from ViTs.\n\nThe paper could benefit from further analysis of why certain temporal attention blocks are more effective (e.g., visualization of learned temporal relations).\n\nSTAVEQ2 is not evaluated on long video sequences beyond the SSv2/MVBench scale, which limits its claims for “temporal reasoning at scale.”\n\n There is no mention of confidence intervals or variance across runs. Gains are reported as absolute.", "was_revised": true, "final_justification": "Considering the rebuttal and the reading of the other reviews, I am happy to recommend the same rating as before", "rebuttal": "We thank you for your thorough and positive review, as well as the constructive suggestions. Below, we address the identified weaknesses and answer the questions in detail.\n\n**W1**: Incremental change\n\nArchitecturally, our Stacked Temporal Attention (STA) blocks are designed to be additive and parameter-efficient. This preserves the powerful pre-trained spatial weights of the vision encoder and ensures training stability, making it a more viable approach than training a model from scratch or performing a destructive architectural change. Our main focus in this work is to *systematically demonstrate the necessity of temporal attention in Video-LLMs*, where it has been largely overlooked or dismissed by recent architectural trends. For instance, Qwen2-VL has only focused on developing Multimodal Rotary Position Embedding for temporal understanding, while relying solely on spatial attention without explicit temporal modeling in the vision encoder, as stated in their paper [6].\n\nOur comprehensive results, presented in Subsection 5.5, Table 4, and Appendix Section F, Table 6 of our paper, and further supported by Table 2 in our rebuttal, directly challenge this design paradigm. These results unequivocally show that incorporating explicit temporal attention mechanisms, like our Stacked Temporal Attention, leads to substantial performance improvements on benchmarks that specifically require robust temporal understanding. Table 1 of rebuttal also shows that newer models like **VideoRoPE** and **InternVideo2.5** which focus on different approaches for advancing temporal understanding, can still benefit from this effective method. We will emphasize this positioning more clearly in the camera-ready version.\n\nAnother compelling proof of this is our experiment on the InternVideo2 vision encoder—the previous state-of-the-art (SOTA) model on SSv2, which already incorporates spatiotemporal attention. Our InternVideo2+STA model surpassed this powerful baseline and achieved a new SOTA on SSv2 classification (Subsection 5.2, Table 2). This critical result shows that dedicated, explicit temporal modeling enhances performance even on top of an existing spatiotemporal mechanism.\n\nThis is also reflected in the positive feedback from other **reviewer tYNw**, describing our work as *\"a novel approach that effectively addresses the limitations of current Video-LLMs in temporal understanding.\"*\n\n**W2 & Q1**: Visualization of temporal attentions\n\nWe agree that interpretability is an important aspect. To address this, we provide an attention map visualization in **Appendix E, Figure 1** that compares InternVideo2 with our InternVideo2+STA. This visualization demonstrates that our model more effectively allocates attention to motion-relevant regions. While new figures cannot be included in the rebuttal itself, we will expand this analysis with additional examples in the camera-ready version to further strengthen the interpretability of our approach.\n\n\n**W3 & Q2**: Evaluation on long video sequences\n\nThank you for this question regarding scalability to longer video clips. Our model's scalability is demonstrated on the VideoMME benchmark, where it achieves consistent gains across videos ranging from **11 seconds to over an hour**. The current limitation on maximum video length is not inherent to our design but is imposed by the base Qwen2-VL architecture's hard context limit (approximately 760 frames). In principle, our Stacked Temporal Attention blocks are designed to scale efficiently with input length.\n\n**W4**: Confidence interval and variance\n\nThank you for this question. Our evaluation protocol follows the established standard for large-scale Video-LLM benchmarking, and is consistent with the evaluation practices of recent leading works in the field, including Qwen2-VL. We report results from a single training run due to the prohibitive computational cost. A single run for models of this scale requires hundreds of GPU-hours, making multiple independent runs infeasible.\n\n**Q2**: Temporal attention block placement\n\nIn our current design, we add one temporal attention layer to each transformer block.\n\nTo assess the impact of STA placement, we conducted two additional ablation studies, in addition to those in Appendix B, that varied the number and placement of temporal blocks within the transformer blocks. The results are summarized in Table 1 of our rebuttal, illustrating how different configurations affect the model’s accuracy. As shown, reducing the number of temporal blocks from 32 to 16 seems to decrease performance. Additionally, the placement of 16 temporal blocks—whether distributed uniformly or concentrated in the early layers of the vision encoder—does not result in significant differences in performance. We will add these experiments to Appendix B, Table 1, in our camera-ready version.\n\n**Table 1**\n\n| Model        | Number of Temporal Blocks | Temporal Blocks Placement | Accuracy (%) |\n| :----------: | :-----------------------: | :-----------------------: | :----------: |\n| STAVEQ2 2B| 32 | All Blocks | 76.04 |\n| STAVEQ2 2B| 16                        | Uniform                   | 74.73        |\n| STAVEQ2 2B| 16                        | First Blocks              | 74.97        |\n\n\n\n\n**Q3**: Comparison to other strategies\n\nWhile other lines of research also aim to improve temporal awareness—such as hierarchical token merging in InternVideo2.5-Chat [3] or advanced positional encodings in VideoRoPE [2]—we view these methods as orthogonal and complementary to our approach. To validate this, we conducted new experiments applying our method to these strong baselines. As presented in Table 2 of our rebuttal, we observed consistent performance gains across all models, demonstrating the synergistic value of our approach.\n\n**Table 2**\n\n| Model                     | Comp. | Dir | Int. | Loc. | Seq. | Type | MVBench | VideoMME(wo) |\n| :-----------------------: | :---: | :--: | :--: | :--: | :--: | :--: | :-----: | :----------: |\n| VideoRoPE [1]              | 81.1  | 81.8 | 60.9 | 79.4 | 80.7 | 85.8 | 57.3    | 61.6         |\n| VideoRoPE+STAVE           | **81.9** | **82.9** | **61.8** | **79.9** | **81.3** | **86.3** | **59.2** | **62.5** |\n| InternVideo2.5-Chat [2]    | 91.3  | 88.7 | 82   | 84.8 | 84.7 | 91   | 75.7    | 65.1         |\n| InternVideo2.5-Chat+STAVE | **91.6** | **89.7** | **82.7** | **85.6** | **85.8** | **91.3** | **76.8** | **65.9** |\n\n\n**Q4**: Generalization to other tasks\n\nTo assess generalization to other tasks such as video captioning, we evaluated our STAVEQ2 7B model against its Qwen2-VL 7B baseline on the challenging CaReBench benchmark [4]. CaReBench is specifically designed to test a model's perceptual accuracy by evaluating it across two categories, 'Action' and 'Object', using F1, Precision, and Recall scores to ensure factual correctness and mitigate hallucination. Our results in Table 3 demonstrate that STAVEQ2 exhibits stronger captioning abilities and improves upon the base model without any caption-specific fine-tuning.\n\n\n**Table 3**\n\n| Model               | A-F1  | A-Percision | A-Recall | O-F1  | O-Percision | O-Recall |\n| :-----------------: | :---: | :---------: | :------: | :---: | :---------: | :------: |\n| Qwen2-VL 7B         | 0.34 | 0.47       | 0.27    | 0.34 | 0.55        | 0.25    |\n| STAVEQ2 7B (Ours)| **0.41** | **0.49** | **0.36** | **0.43** | **0.61** | **0.33** |\n\n\nFor moment retrieval, task-specific tuning would likely be required. The attention mechanism itself is general and can integrate into these models.\n\n**Q5**: Isolating the effect of using 1D RoPE vs. 2D or the inclusion of LoRA adapters during training\n\n1. In our architecture, 2D RoPE is applied to the spatial attention layers and 1D RoPE is applied to our separate temporal attention layers. As these embeddings operate on orthogonal dimensions (space and time), a direct \"A vs. B\" ablation is not straightforward. We would be grateful if you could clarify the specific experiment they envision so we can provide a more detailed response.\n\n2. We isolated the effect of LoRA through a direct comparison on the SSv2 benchmark. We first confirmed that the SOTA InternVideo2 baseline could not be improved further with either LoRA or full fine-tuning. However, when we introduced our architectural change by adding STA (InternVideo2+STA), the model surpassed this baseline and achieved a new SOTA (Subsection 5.2, Table 2). This comparison clearly demonstrates that *the significant performance gain is attributable to our STA module, not the training method.*\n\n**References**\n\n[1] Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution, Peng Wang et. al., arXiv preprint arXiv:2409.12191 2024.\n\n[2] VideoRoPE: What Makes for Good Video Rotary Position Embedding?, Xilin Wei et al., International Conference on Machine Learning 2025.\n\n[3] InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling, Yi Wang et. al., arXiv preprint arXiv:2501.12386 2025\n\n[4] CaReBench: A Fine-Grained Benchmark for Video Captioning and Retrieval, Yifan Xu et. al., arXiv preprint arXiv:2501.00513 2025."}, {"review_id": "9nA4B7gXKU", "reviewer": "Reviewer_rKct", "summary": "This work focuses on improving the handling of temporal information by adopting stacked temporal attention in vision encoders. The motivation for breaking down the current Large Vision Language Model (LVLM) seems solid and reasonable to readers. However, it is somewhat challenging to fully understand why spatial and then stacked temporal works better than the spatiotemporal approach. It would be great if the author presented a bit more theoretical or empirical details.", "questions": "[Major]\nI failed to catch why adopting 'stacked temporal attention' is the best approach over the rest. I understand that it could help the model handle the temporal dimension more efficiently, but I wonder whether it is the clear winner for the general cases. \n1. It would be great if they could provide a bit more theoretical or empirical details. For example, in Table 1 in Supplementary Section B, I believe it would provide a clearer comparison if the table also included spatiotemporal information. \n2. It would be more powerful if they provided a similar experiment over the pretrained InternVideo model (by replacing their spatiotemporal attention).\n3. Similarly, it would be even more powerful if they performed the InternVideo experiment (= adding rows) in Tables 3 and 4. \n\n\n[Minor]\n1. Based on the code attached in the supplementary material, I believe the author will release the data, not just the attention implementation. However, I haven't seen a clear statement about data in their manuscript. It would be even better if they also mentioned the license along with the (curated/processed) datasets.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Great analysis of the current limitation on visual understanding of LVLM.\n2. Following problem identification and their approach also sounds reasonable.\n3. This manuscript compares their performance across multiple models, including some ablations.\n4. Equations and figures are promptly used. It helped the reader understand very well.\n\nWeaknesses:\n1. It is somewhat challenging to clearly discern the author's approach, as it does not fully incorporate theoretical or empirical details explaining why this approach would perform better than spatiotemporal attention.\n2. In line with the previous point, it is a bit unclear why InternVideo (which incorporates spatiotemporal attention) is not included in the main tables, Tables 3 and 4 (but is instead included in Table 2 only).\n3. I failed to see the authors' plan for open-sourcing the data (and their license).", "was_revised": true, "final_justification": "Thank the authors for their detailed response. I read the whole review and the rebuttal response fairly addressed my concerns, although some of them sound a bit indirect to me. For example, InternVideo2 + STA may introduce more parameters, and it would be much clearer if they could provide more apple-to-apple comparisons by replacing the spatiotemporal and introducing their STA with finetuning. Nonetheless, as I got understandable responses for most of the main questions, I would like to stick to my current score. I hope this rebuttal strengthens their manuscript.", "rebuttal": "Thank you for your detailed feedback and questions. We are encouraged by the positive assessment of our work, and we address each of your concerns below and provide clarifications, additional results, and plans for the camera-ready version.\n\n**W2 & Q3:** Absence of InternVideo2-Chat in Tables 3 and 4\n\nThanks for pointing this out. This is due to a fundamental architectural limitation of the public InternVideo2-Chat model: it is restricted to processing a maximum of 8 input frames in each prompt. This is fine for SSv2 dataset which has video length of 2 to 6 seconds, but we cannot evaluate it on benchmarks that have longer video lengths like VideoMME (ranging from 11 seconds to over an hour). Therefore, a direct and fair comparison on those specific tables was not feasible. We will add a clear statement to the camera-ready version of the paper explaining this limitation. We will perform comparisons on suitable short-context benchmarks.\n\n**W1 & Q[Major]** Theoretical and empirical justification for Stacked Temporal Attention (STA)\n\nThanks for these crucial questions. Our approach is grounded in empirical validation and clear architectural advantages. As demonstrated across our comprehensive results (Subsection 5.5, Table 4; Appendix F, Table 6; Rebuttal, Table 1), adding STA to vision encoders consistently improves performance, proving it is superior to relying on spatial-only attention.\n\n**Table 1**\n| Model                     | Comp. | Dir | Int. | Loc. | Seq. | Type | MVBench | VideoMME(wo) |\n| :-----------------------: | :---: | :--: | :--: | :--: | :--: | :--: | :-----: | :----------: |\n| VideoRoPE[1]              | 81.1  | 81.8 | 60.9 | 79.4 | 80.7 | 85.8 | 57.3    | 61.6         |\n| VideoRoPE+STAVE           | **81.9** | **82.9** | **61.8** | **79.9** | **81.3** | **86.3** | **59.2** | **62.5** |\n| InternVideo2.5-Chat[2]    | 91.3  | 88.7 | 82   | 84.8 | 84.7 | 91   | 75.7    | 65.1         |\n| InternVideo2.5-Chat+STAVE | **91.6** | **89.7** | **82.7** | **85.6** | **85.8** | **91.3** | **76.8** | **65.9** |\n\n\nTo provide stronger evidence, we tested STA on the InternVideo2 vision encoder, which was explicitly trained on the Something-Something v2 (SSv2) dataset and held the previous state-of-the-art (SOTA) result. Our InternVideo2+STA model surpassed this powerful baseline and achieved a new SOTA on the SSv2 classification (Subsection 5.2, Table 2). This empirical evidence is critical, as it shows that dedicated temporal attention enhances the model’s ability to interpret temporal dynamics *even when added on top of an existing spatiotemporal attention mechanism*, enforcing more explicit modeling. An example attention map visualization of InternVideo2+STA is presented in Appendix E, Figure 1, which shows that our model allocates more attention to motion-relevant regions.\n\nFurthermore, the STA module is **additive**, preserving the powerful pre-trained spatial weights of the vision encoder. In contrast, converting a baseline to use spatiotemporal attention is a **destructive** process. For instance, modifying Qwen2-VL would require replacing 2D RoPE with 3D RoPE, which disrupts pre-trained weights and necessitates extensive retraining. Our approach elegantly bypasses this problem, demonstrating a clear advantage in performance and efficiency.\n\nRegarding whether STA is the \"clear winner\" in all general cases, our results confirm its superiority for tasks requiring fine-grained temporal reasoning.\n\n**Q1:** Adding spatiotemporal attention to Appendix B, Table 1\n\nModifying Qwen2-VL’s spatial attention in its vision encoder to incorporate spatiotemporal attention involves replacing 2D Rotary Position Embedding (RoPE) with 3D RoPE. This change disrupts the pre-trained weights and requires extensive retraining. In contrast, the STA module is additive, allowing us to preserve the powerful pre-trained spatial weights of the vision encoder, resulting in a more robust training process which is unfortunately beyond the computation resource budget that is available to us. In Appendix B, Table 1, results for training STAVEQ2 and Qwen2-VL on SSV2-T10 dataset are presented. However, this dataset alone is insufficient for training the vision encoder to adapt to new position embeddings.\n\n**Q2** Experimenting over InternVideo2 by replacing their spatiotemporal attention\n\nReplacing the core spatiotemporal attention in the InternVideo2 vision foundation model with spatial-only attention would remove its primary mechanism for temporal reasoning, significantly limiting its capacity for video understanding. Instead, we chose to explore a more complementary direction by integrating our STA module into InternVideo2's existing spatiotemporal attention. The results of this experiment are presented in Subsection 5.2, Table 2.\n\n**W3 & Q[Minor]:** Plan for open-sourcing the data and their license\n\nThanks for bringing this to our attention. We would like to clarify that all source datasets used are publicly available, and we have provided citations. Importantly, we will release our complete data processing pipeline and the implementation code for our models, as these have already been included in the supplementary materials.\n\nWe hope these clarifications and new results have addressed your concerns. We are confident that these additions will significantly strengthen the paper.\n\n**References**\n\n[1] VideoRoPE: What Makes for Good Video Rotary Position Embedding?, Xilin Wei, Xiaoran Liu, Yuhang Zang, Xiaoyi Dong, Pan Zhang, Yuhang Cao, Jian Tong, Haodong Duan, Qipeng Guo, Jiaqi Wang, et al., International Conference on Machine Learning 2025.\n\n[2] InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling, Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Cheng Ma, Hong Huang, Jing Gao, and Meng Dou, arXiv preprint arXiv:2501.12386 2025."}, {"review_id": "TqrhQmIEwj", "reviewer": "Reviewer_tYNw", "summary": "The paper presents STAVEQ2, a novel Video-LLM architecture that incorporates stacked temporal attention modules within the vision encoder to enhance temporal understanding in video question answering tasks. The authors identify critical limitations in current Video-LLMs in capturing fine-grained temporal dynamics and propose an improved model that addresses these shortcomings. The paper demonstrates significant performance improvements on several benchmarks, including VITATECS, MVBench, and Video-MME, with gains of up to +5.5%. Ablations confirm STA’s efficacy, and application to video-only models (e.g., InternVideo2) yields new SOTA on SSv2 action recognition (78.0%).", "questions": "1. How many frames were used per video? Were frames uniformly sampled or dynamically selected?\n2. The authors should explicitly address SSv2-T10 dataset statistics?\n3. Can the author explain the number of transformer blocks in the experiment? And add the ablation experiment of number of transformer blocks.\n4. Can the author add comparisons to VTG-LLM, InternVideo2.5, and VideoPrism? \n5. Should the authors provide clearer motivation, focus their problem statement more precisely, and adequately address the other concerns raised, I would consider raising my score accordingly.", "limitations": "yes", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths \n1. The introduction of stacked temporal attention modules directly within the vision encoder is a novel approach that effectively addresses the limitations of current Video-LLMs in temporal understanding. This design explicitly equips the vision encoder with temporal attention, enabling better capture of action progression and frame relationships.\n2. The paper provides extensive experiments across multiple benchmarks, demonstrating the robustness and general applicability of the proposed method. The results show significant improvements over recent state-of-the-art Video-LLMs, highlighting the effectiveness of the temporal attention modules.\n3. The authors conduct a thorough analysis of the limitations of current Video-LLMs, providing valuable insights into their shortcomings in temporal reasoning. This analysis motivates the proposed approach and underscores the importance of temporal modeling in video understanding tasks.\n4. The proposed temporal attention modules are parameter-efficient, using up to four times fewer attention heads than spatial attention while maintaining head dimension. This design minimizes computational overhead while significantly enhancing temporal modeling.\n\nWeaknesses.\n1. All experiments omit the number of frames per video used for evaluation, hindering reproducibility and fair comparison.  This must be clarified.\n2. The SSv2-T10 dataset’s class distribution, train/test splits  are undefined (Sec 3.1), and Appendix fails to clarify the list of selected action classes.\n3. While Appx.B tests STA placement/head scaling, critical ablation of Impact of STA depth (number of transformer blocks) is missing. \n4. Recent Video-LLMs (e.g., VTG-LLM[1], Internvideo2.5[2],InternVL3[3]) and temporal modeling methods (e.g., VideoPrism[4]) are absent from comparisons.  This omission undermines claims of architectural superiority.\n5. The Introduction lacks sufficient clarity. The challenges presented are too vague. Temporal understanding is a broad-ranging task; the paper should focus on addressing a specific aspect of it. Proposing a method to solve temporal understanding in its entirety is impractical. The motivation needs to be significantly more concrete.\n\n[1] Guo Y, Liu J, Li M, et al. Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding[C]//Proceedings of the AAAI Conference on Artificial Intelligence. 2025, 39(3): 3302-3310.\n\n[2] Wang Y, Li X, Yan Z, et al. InternVideo2. 5: Empowering Video MLLMs with Long and Rich Context Modeling[J]. arXiv preprint arXiv:2501.12386, 2025.\n\n[3] Zhu J, Wang W, Chen Z, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models[J]. arXiv preprint arXiv:2504.10479, 2025.\n\n[4] Zhao L, Gundavarapu N B, Yuan L, et al. Videoprism: A foundational visual encoder for video understanding[J]. arXiv preprint arXiv:2402.13217, 2024.", "was_revised": true, "final_justification": "I will maintain my original score. While I appreciate the clarifications provided, several critical concerns remain unaddressed. \n\nRegarding W1 & Q1, the response explicitly details frame sampling only for Qwen2-VL and InternVideo2, but fails to specify the frame sampling strategy or counts for other baseline models evaluated in the main paper.\n\nConcerning W2 & Q2, the rebuttal states 13,984 samples in SSv2-T10 dataset.  However, this contradicts the main paper's assertion of 14,462 samples (Section 3.1). \n\nFor W4 & Q4 and Table 2, I was unable to reproduce the reported results for MVBench and VideoMME on InternVideo2.5. To validate these claims, please provide the exact experimental configuration: (1) the specific codebase/library versions used (e.g., PyTorch, Transformers, lmms-eval, VLMEvalKit), (2) config files, sampling strategies, input resolution, and batch size. Without this granularity, the reproducibility and validity of Table 2’s results remain questionable.", "rebuttal": "We thank you for the thoughtful and detailed comments. We address each concern below and provide clarifications, additional results, and plans for the camera-ready version.\n\n**W1 & Q1:** Number of frames per video clip\n\nThank you for pointing this out. For most experiments, we sample frames at **a fixed rate of 2 fps** across the entire video, which is the default for the Qwen2-VL architecture. This dynamic approach ensures the number of frames scales with the video's duration. For instance, a standard 10-second clip from MVBench yields 20 frames, while shorter SSv2 videos (typically 2-6 seconds) result in 4-12 frames. For InternVideo2, we sample exactly 8 frames per video to comply with its fixed 8-frame input requirement. We will ensure these details are explicitly stated in the camera-ready version.\n\n**W2 & Q2:** SSv2-T10 dataset statistics\n\nThanks for highlighting this point. For zero-shot experiments, we used a total of **3392 samples**, and for training on the **SSv2-T10** subset, we had **8476 training** and **2116 testing** samples. The distribution of the classes in all the experiments is as follows:\n\n - Pulling [something] from left to right – 14.68%\n\n- Pulling [something] from right to left – 14.97%\n\n- [Something] falling like a rock – 13.12%\n\n- Picking [something] up – 9.26%\n\n- Throwing [something] in the air and letting it fall – 7.31%\n\n- Throwing [something] in the air and catching it – 8.90%\n\n- Moving [something] away from [something] – 8.61%\n\n- Moving [something] closer to [something] – 8.55%\n\n- Rolling [something] on a flat surface – 11.85%\n\n- Poking a stack of [something] so the stack collapses – 2.74%\n \nWe will add the full class list, distribution, and split details in the Appendix.\n\n\n**W3 & Q3:** Missing ablation on STA depth\n\nWe appreciate this suggestion. Since the computational complexity of temporal attention is lower than that of spatial attention—because the number of frames is smaller than the number of patches in each frame—we found no compelling reason to limit these layers to only certain transformer blocks. However, they could conceptually be placed adaptively across the transformer blocks in the vision encoder.\n\nTo assess the impact of STA depth, we conducted two additional ablation studies, in addition to those in Appendix B, that varied the number and placement of temporal blocks within the transformer blocks. The results are summarized in Table 1 of our rebuttal, illustrating how different configurations affect the model’s accuracy. As shown, reducing the number of temporal blocks from 32 to 16 seems to decrease performance. Additionally, the placement of 16 temporal blocks—whether distributed uniformly or concentrated in the early layers of the vision encoder—does not result in significant differences in performance.  We will add these experiments to Appendix B, Table 1, in our camera-ready version.\n\nTable 1\n\n| Model        | Number of Temporal Blocks | Temporal Blocks Placement | Accuracy (%) |\n| :----------: | :-----------------------: | :-----------------------: | :----------: |\n| STAVEQ2 2B| 32 | All Blocks | 76.04 |\n| STAVEQ2 2B| 16                        | Uniform                   | 74.73        |\n| STAVEQ2 2B| 16                        | First Blocks              | 74.97        |\n\n\n**W4 & Q4:** Need for more baselines\n\nThank you for this crucial point regarding comparisons to recent state-of-the-art models. To address this, we conducted two additional experiments for this rebuttal, using two highly competitive baseline models, **InternVideo2.5 [1]** and **VideoRoPE [2]**, and applied our STAVE method to them. These models represent distinct and important lines of work aimed at improving temporal understanding. InternVideo2.5 utilizes architectural innovations like hierarchical token merging, while VideoRoPE focuses on enhancing positional encoding. As shown in Table 2 of our rebuttal, applying STAVE consistently improves the performance of both models across the benchmarks. This result suggests that our approach is orthogonal and complementary; it enhances the vision encoder's core temporal processing in a way that provides additive benefits to models with different architectural strengths.\n\nRegarding the other models you mentioned, VTG-LLM[3] is highly specialized for temporal grounding, which differs significantly from our focus on general video question-answering, making direct comparison less meaningful. VideoPrism[4] is a vision-only foundation model, complicating a straightforward comparison with our full Video-LLM. Finally, InternVL3[5] was released very shortly before the submission deadline and employs test-time scaling techniques that complicate a fair comparison. While we aimed to be thorough, we must also note that our limited computational resources make it infeasible to benchmark against every recently published architecture.\n\nTable 2\n\n| Model                     | Comp. | Dir | Int. | Loc. | Seq. | Type | MVBench | VideoMME(wo) |\n| :-----------------------: | :---: | :--: | :--: | :--: | :--: | :--: | :-----: | :----------: |\n| VideoRoPE[1]              | 81.1  | 81.8 | 60.9 | 79.4 | 80.7 | 85.8 | 57.3    | 61.6         |\n| VideoRoPE+STAVE           | **81.9** | **82.9** | **61.8** | **79.9** | **81.3** | **86.3** | **59.2** | **62.5** |\n| InternVideo2.5-Chat[2]    | 91.3  | 88.7 | 82   | 84.8 | 84.7 | 91   | 75.7    | 65.1         |\n| InternVideo2.5-Chat+STAVE | **91.6** | **89.7** | **82.7** | **85.6** | **85.8** | **91.3** | **76.8** | **65.9** |\n\n\n**W5:** Introduction Clarity and Motivation\n\nThank you for this feedback. While other reviewers noted that our problem is \"well-identified\" (**Reviewer q8eF**) and that we provide a \"great analysis\" of current limitations (**Reviewer rKct**), we want to clarify, that by temporal understanding, we are not addressing the topic “in its entirety”. Our work focuses specifically on video question answering (Video QA) tasks, particularly those that are temporally challenging. Our motivation stems from the observation that while current Video-LLMs are effective generalist models, they often struggle with questions requiring a deeper comprehension of temporal relationships and actions within the video. Our central hypothesis is that this failure is due to a lack of explicit temporal modeling mechanisms within the vision encoders of these models. To address this diagnosed weakness, we introduce STAVEQ2 and validate the hypothesis by showing significant performance gains over the baseline across all the benchmarks. We show this method's general applicability and consistent improvement over the baselines by subsequently it to other modern Video-LLMs.\n\n\n**References**\n\n[1] InternVideo2.5: Empowering Video MLLMs with Long and Rich Context Modeling, Yi Wanget. al., arXiv preprint arXiv:2501.12386 2025.\n\n[2] VideoRoPE: What Makes for Good Video Rotary Position Embedding?, Xilin Wei et. al., International Conference on Machine Learning 2025.\n\n[3] Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding,  Guo Y, Liu J, Li M, et al., Proceedings of the AAAI Conference on Artificial Intelligence. 2025\n\n[4] Videoprism: A foundational visual encoder for video understanding, Zhao L, Gundavarapu N B, Yuan L, et al., arXiv preprint arXiv:2402.13217, 2024.\n\n[5] Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models, Zhu J, Wang W, Chen Z, et al., arXiv preprint arXiv:2504.10479, 2025."}]

Author rebuttal:

Meta-review: The final review scores were two "Accepts," one "Borderline Accept," and one "Borderline Reject."

All three positive reviewers explicitly stated that the author rebuttal sufficiently addressed their concerns. One reviewer subsequently raised their rating, and another maintained their stance for acceptance even after considering the negative review.

The negative reviewer, however, chose to keep their original rating following the post-rebuttal discussion. The main points of concern raised were: 1) the use of a different FPS in some baseline models, 2) minor calculation errors, and 3) the reviewer's failure to reproduce some results.

The AC has evaluated these concerns and finds them to be minor factors for the following reasons:
- Fair Comparison: The primary baseline model, which the proposed model is built upon and is its best-performing competitor, uses the same FPS, ensuring a fair comparison.
- Minor Mismatch: The calculation mismatch in the dataset is a minor error that does not negate the paper's overall validity.
- Reproduction Issues: The reproduction failure occurred on one setup with a single model, which the authors were only asked to include during the rebuttal period. Furthermore, the reviewer did not use the authors' code but their own reproduction efforts. This failure cannot negate the paper's overall validity.

Given the overall positive consensus and the minor nature of the concerns raised by the dissenting reviewer, the AC believes the paper meets the NeurIPS standard and recommends acceptance.
Decision: Accept (poster)