Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames
Authors: Anurag Arnab
Ahmet Iscen
Mathilde Caron
Alireza Fathi
Cordelia Schmid
Venue: NeurIPS 2025 poster ()
Keywords: VLM
Long-video understanding
video question-answering
long-context
Primary area: applications
TL;DR: We propose an inference-strategy for long-video QA, which substantially improves the accuracy of a VLM by curating its input context.

Abstract: Despite recent advances in Vision-Language Models (VLMs), long-video understanding remains a challenging problem. Although state-of-the-art long-context VLMs can process around 1000 input frames, they still struggle to effectively leverage this sequence length, and succumb to irrelevant distractors within the context window. We present Dynamic Context Aggregation, an inference strategy for video question-answering that curates the model's input context. We use the VLM itself to iteratively identify and extract the most relevant frames from the video, which are then used for answering. We demonstrate how leveraging more computation at inference-time to select the most relevant context leads to improvements in accuracy, in agreement with recent work on inference-time scaling of LLMs. Moreover, we achieve state-of-the-art results on 4 diverse video question-answering datasets, showing consistent improvements with 3 different VLMs. In particular, our method shines on longer videos which would not otherwise fit in the model's context window: On longer videos of more than 1 hour on LVBench, our approach using a context window of 32K outperforms the same VLM using standard inference with a 700K context window by 2.8 points.

Temporal Chain of Thought: Long- Video Understanding by Thinking in Frames
Anurag Arnab" Ahmet Iscen"
Mathilde Caron Google DeepMind
Alireza Fathi
Cordelia Schmid
Abstract
Despite recent advances in Vision-Language Models (VLMs), long-video under- standing remains a challenging problem. Although state-of-the-an long-context VLMs can process around 1000 input frames, they still struggle to effectively lever- age this sequence length, and succumb to irrelevant distractors within the context window. We present Temporal Chain of Thought, question-answerrig that curates the model's input context. We use the VLM itself to iteratively identify and extract the most relevant frames from the video, which are then used for answering. We demonstrate how leveraging more computation at petrence-timme to select the most relevant context leads improvements in accu- racy, in agreement with recent work on inference- time scaling of LLMs. Moreover, achieve satte-of-thh-ar results 4 diverse video showing consistent improvements with different VLMs. In particular, our method shines longer videos which would not otherwise fit within the model's 's context window: On longer videos of more than hour on LVBench, our approach using a context window of 32K outperforms the same VLM using standard inference with a 700K context window by 2.8 points.
inference strategy for video
datasets,
Introduction
Despite recent advances in Vision Vision-Language Models (VLMs) 6 32 34 55], understanding long videos remains challenging problem. This difficulty stems from the fact that this task requires a VLM process long sequence of input tokens, and requires the model to possess host of interrelated abilities including action and scene understanding, long-term memory, and tracking state changes and interactions among others. The long-context ability of leading VLMs, that enables models to process hundreds or even thousand frames of input context [36] 55| 56), is a valuable step forward this regard. However, numerous studies have shown that processing longer contexts can saturate or degrade accuracy, as the model is overwhelmed with irrelevant or misleading content 223
28 37
Based the observation that too large of an input context be distracting, propose inference strategy, Temporal Chain Thought, which first aggregates relevant context from the input video, Prior works based on the principle of removing 61 62 used an ensemble of multiple models, typically using one model caption individual frames, another to find relevant ones, and finally answer the question with an L.M.. In contrast, we use only a single VL. to both select the relevant context, and to answer the question, and show how our inference strategy provides substantial improvements.
and then it answer the question (Fig.
distracting context from a video (28 45
Our approach is motivated by recent studies in Large Language Models (LLMs) which suggest that scaling inference-time time computation is more effective than scaling the number of model parameters 99
20 52 65 Similarly, we show how leveraging more computation to aggregate relevant information from the video results in higher accuracy. In addition, approach iteratively extracts relevant
Equual contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

nidutvideo-- + question
Temporal Chain of Thought (ours)
VLM Answerer
VLM: Context 3g gregation
divide then by species
lus Iusifcatoo:
Justflotton:
In Fram 100, she 1i cakknng the siovera out Jie vase and proper Lng co acconge thes...
Ptems 181-197: ale io duparatina the t-rsers,, tite indscates that she wante to dovaide then according to acecten....
VLM: Answerer
Mhy doas the vi viogger take the fiewers out of bottle?
Do replace them
X
Standard inference (b (baseline)
Figure Temporal Chain of Thought. Motivated by the fact that long input contexts can have distractors which confuse the model, we use the VLM itself to first extract relevant context (blue box) before processing it. Our approach improves accuracy, and by iteratively processing parts of the video time, can also reduce the model's 's required context window.
context from the video, it means that we can effectively process videos that would not otherwise fit within the model's context limit. Moreover, our approach has connections Chain-of- ohoouhh prompting (63] (and multi-step extensions of (59 68 language, where the model is prompted to first output textual "thoughts" which help to produce the final answer. As we aggregate the relevant frames in the vidco, we can think of these frames as "visual thoughts". Furthermore, as by-product, can use the model's justifications for choosing relevant frames to interpret it (Fig. I.
We find that the general principle of aggregating relevant context from video is beneficial for video answering (QA), with our proposed method consistently improving results across 4 datasets and different VLMS. For shorter videos, on the order of hundreds of frames, our inference strategy improves results even when the entire video could fit within the model's s context window, emphasising that by removing distractors from the input, we can improve the model's s reasoning ability.
For longer videos, on the order of a thousand frames, such as LVBench [58], our method shows even larger improvements. Given a fixed context-window budget for VLM, our model can iteratively extract the most relevant context leading to substantial accuracy gains. Furthermore, our inference strategy focuses the model on the most relevant context, we can even outperform standard, baseline inference with much longer context window.
In summary, we propose the following contributions:
A novel VLM inference strategy for video QA.
Thorough experimental analyses confirming that the principle of context aggregation is effective, that our approach outperforms standard long-context inference across a range of computational budgets, is adaptive the question type and generalises to multiple VLMs. State-of-the-art results on 4 video understanding benchmarks. In particular, on LVBench where videos average 68 minutes in length, we improve by 11.4 points given context- window budget of 32K tokens. Moreover, our iterative approach using a 32K context-limit outperforms long-context baseline using the same 700K total tokens by 2.8 points.
2 Related Work
Our work is motivated by several prior studies, predominantly in the domain of natural language, which have demonstrated how Large Language Models (LLMs) are not able to
Long-context LLMs
70 74). By asking questions where the position of crucial piece of information in the input context is varied in controlled manner, studies have shown that performance degrades considerably when the relevant context not at the beginning end of the imput sequence [37
effectively leverage their full input contexts 233 37 64
67
64 77].
Long-context video understanding
The most related works for long-video understanding, however, 61 62 72). These works represent long video by first computing captions at each frame of the video with a dataset-specifie model 22 35| 73). which are then fed an LLM along with the input question to produce an answer. Observing that redundant captions degrade the performance of the "answerer" LLM, a number of strategies have been proposed: Video Agent [6]] begins from uniformly sampled frames, and uses EVA-CLIP embeddings frames until sufficient context has been obtained for GPT-4 to output an answer confident in. Video Tree (62 in contrast clusters per- permframe captions together (where each cluster is then represented
are [5] 22 28 45 57
[53] to iteratively retrieve

---

by the caption of the frame closest to the centroid), and then only uses the top-scoring centroids in the final answering phase. Language Repository 28 aggregates per- per-frame captions into global textual representation of the video, using an L.LM to summarise different captions together, and CLIP similarities to prune redundant captions. VGdeoRAG [39], in contrast, supplements the video with auxilliary model outputs [2 27 46 50], namely object detection, ASR and OCR, represented as text. Similarly, [66 42 54
call external tools based on per-frame captions to answer questions.
Although we also extract relevant context from the input video, our approach is fundamentally different in that we operate directly on video frames, and not captions as an intermediate representation. As we do not rely on initial per-frame captions, our approach is not limited by the captioner missing details relevant to the question (particularly because the captioning these works not conditioned on the input question). Moreover, we use only a single VLM in the entire inference process unlike the aforementioned works, which means that our approach is conceptually more elegant and simpler to deploy. The fact that we use a single model to generate intermediate outputs (the relevant frame indices to the question) is akin to "Chain-of- Thoouhhh 29 Chain-of-Thoughh
prompting.
Chain-of-Thoughh (CoT) 63 was originally developed for few-shot prompting of LLMs for symbolic reasoning arithmetic tasks. Concretely, the model is prompted to first output (in natural language) the steps first required to solve a reasoning problem, before outputting the final answer. This approach is effective as the LLM is conditioned on the initial reasoning (or "thoughts"") before making its final prediction during autoregressive decoding. Subsequent works extended this approach, proposing inference-scaling strategies involving multiple LLM calls: The LLM first produces multiple hypotheses, which are then ranked, and the top-scoring ones explored further 43 [2 [5 (68) Our method has similarities [63] but instead of producing "thoughts"
as language, predicts relevant frame indices in the video instead. Note that for video, Fei el al. [17 first predict spatio-temporal scene graphs (26) and use these intermediate representations for answering the question. Our method is more general, as it is not object-centric like 117, and our iterative approach for finding the most relevant frames has analogues of [52
59
68
for video understanding.
Token- or frame-selection Finally, numerous prior works learn neural network layers to reduce the number of tokens that need to be processed by a subsequent transformer X
[4
25
48
67
75
Among
these, models specialised for video typically learn to select individual frames 10
30
3
49
60
However, these methods need to backpropagate gradients through the subsequent transformer during training, meaning that is not computationally feasible to employ such approaches with the largest, pretrained VLMs 1 551 as our work. Such models are also trained for fixed, small number of input frames (i.e. 32 for SeViL (69 and ViL.A [60]), whilst we show that our approach can handle thousands. Morcover, perhaps surprisingly, we demonstrate that we do not need separate networks to select relevant frames, but can use the VLM itself to do so in an effective and adaptive manner.
Proposed Approach
We begin by reviewing the standard inference process for Vision Language Models (VLMs) before describing our proposed Temporal Chain of Thought.
3.1 Standard VLM inference
The standard inference approach for VLMs to answer a question, q. about an input video, x E RT*HxWX is to simply forward it through the model, f.
a a= f(, q).
(1)
where f denotes the VLM, T, H and W denote the temporal- and spatial dimensions respectively, and a the predicted answer, where q and a are sequences of language tokens which index a discrete vocabulary, V. Note that visual inputs, x, are typically projected, or tokenised into the same space as language tokens. As the overall sequence length of the model is limited by computation, the frames of the video typically need to be subsampled to fit within the context-limit, k. of the model. Current models with the longest context windows can typically fit videos of up to one hour at fps [36 35)
3.2 Temporal Chain of Thought
Our method is motivated by the fact that although VLMS are now capable of handling increasing large input context lengths R6 (55] (56) they often still struggle to leverage this effectively and are confused by irrelevant distractors within this large context [23] 28 37).
3

---

Single Stage Temporal Thought
SegantieSemment Temporal of Thought
ViDOo MENEEL
agg
enlectiion proomee:
( SA seggoonts
Single-Stage
TCoT
Prisee
Zist of Ecasn iddt: [R. Flcatiin
Fcene 2:
da dhe zigit Lren plate
Frane
steas ace the
Single-Stage TCoT
Queattion
Figure 2: Temporal Chain of Thought. We use Single-Step TCoT (left, Sec. 3.2) to construct our final approach (right). Namely, we use the VLM itself to extract relevant frames from an input video clip. conditioned on the imput question. To scalably process longer videos, we perform this approach within I segments which span the video to extract the most relevant context, Finally, we use only the extracted context for answering.
Therefore to answer question, q. about an input video, x, we do not directly pass both inputs to the model directly. Instead, we decompose video guestion-answerde context, from the input x, and then answering using instead (Fig. D.. Crucially, this decomposition performed by the same instruction-tuned VLM which will perform the subsequent answering, drawing inspiration from Shain-of-Thoughhe and related LLM inference strategies [52 63 68).
into first extracting the relevant
Formally, our inference procedure consists of stages: First we assemble the relevant context, from the input video x and question q. Thereafter, we answer the question. We denote these stages as
G q)
H(c, q),
where we denote G the context aggregation function, and H f(c, q) is the answering function, which simply forwards the extracted context through the VLM.
Note that G itself can be a multi-step inference process. It is also adaptive in that the number of context tokens, that are selected is dept-deeendent as long as it fits within the VLM's context limit, k. Importantly, we use the same VLM for both G and H and do not rely on any external models or tools. Note that by carefully designing G. we can also effectively process long videos which would otherwise not fit the model's context limit.
We first introduce a simple form of next, Single-Step Temporal Chain of Thought (TCoT), which construct our final approach, Dynamic-Segment TCoT.
Single-Step TCoT In this simple approach (Fig. 2 left), which is the basis of our final method, we simply query the VLM for what frames it needs to answer a question. Given up N frames from the input video, x [.i,.... N. which fit the context-limic of the model, we prompt the model to output the frame ids which are relevant to answering the given question as
S, = S(x, q)
where denotes the VLM selection
call (prompt in Fig. 3 S the selected frames, and a textual justification of this decision, which can be used for interpreting the model's predictions (Fig. D. x is the resampling of the in- put x using the frame ids in S, namely (z.,...... J for i, .,....J S.
You will be given a question about a video and five possible answer options.
FrameID 1:[frame1),.
FrameeI FrameID [frame N)
Question: (question) Possible answer choices: (answer choices)
Return the frame ids which can answer the given question. Please use the following JSON format for your output: ('frame ids': [List of integer frame IDs], 'justification': Justification about your output)
We validate S to ensure that the frame indices are in ascending order, contain no duplicates and are within bounds. For simple parsing, we prompt the model to predict in the JSON for- mat [15), and for responses that fail to parse, we assume that = [1....,N] N] all frames within the video.
Figure 3: Prompt for VLM selection call, S, (Eq. 4).

---

Additionally, in practice, we found that the relevant frames, x are sometimes too succinct, which may make it more difficult for the answerer, H, to answer from alone. For example, as shown in App. C for the question, On what floor is the washing machine?, the VLM correctly localises the washing machine, but is otherwise difficult to discern which floor of the house is on. To remedy this issue,
we also include a small number, of uniformly sampled frames from the original input, x, denoted as X/u.. where u < N. Thus, the final context aggregated is с = & U X/i--
This simple approach removes redundancy in the input x which may otherwise confuse the model 100 28 37 69]. However, it does not scale to long input sequences. This is because we need to simultaneously process all frames, which may exceed the model's 's context limits, and this subtask of finding relevant frames in a long video is itself prone distractors. We address these issues next.
Dynamie-Segment TCoT To decouple the video length from the context limit of the model, and to overcome limitations frame recall from uniformly sampling input frames to fit within the model's context limit, we design Dynamic-Segment TCoT. As shown in Fig. 2 we partition the videos into ! separate segments, which we process independently. before aggregating them to form a holistic video representation.
To process the input video x, comprising N frames, we divide it into / non-overlapping segments of equal length. We denote each segment as X; {f((-1)m++:-... Fim) J. where E (1....,I} indexes the segment, and m = //l represents the number of frames per segment.
Note that we may have a large segment size, m when either the video is long (large N) or few segments (small 1). This presents a challenge as firstly, the segment length may exceed the model's 's context limit, k. Second, larger segments may contain more distractors, making it harder to identify relevant frames. To mitigate these issues, we uniformly sample 8 frames from each segment x;, denoted as X This ensures a tractable number of frames are considered, balancing computational cost with temporal coverage. The sampled frames from each segment are then processed independently, and the resulting outputs concatenated to form the final representation:
* = [ 5(x4,) 9)--
(5)
As we processed segments independently, it is possible that & is larger than our context-window limit, k. Moreover, we also find it beneficial include coarse, uniform context, X(1) as before. Therefore, if & contains more frames than k, we refine x by uniformly sampling m = k - u frames from it, and adding u uniform context frames. Therefore, the final context aggregated is c 8/mi U|ul]-
Discussion Partitionnng a video into segments ensures that we can process a long-video with a fixed computational cost, regardless of the video-length, N. For standard VLM inference, the cost grows with the length of the video, and is limited by the maximum supported context-limit, k. In contrast, the required context-length with our approach is always fixed at s, to process a total of I frames. Note that our method not only decomposes a video into smaller segments, but importantly reasons across them in the answering stage, analysed in Tab. of AA
Moreover, by varying the number of segments, we can smoothly increase both inference- computation and accuracy (which we show experimentally in the next section). These trends are in agreement with recent work in language p 59] which also uses additional compute inference time to solve challenging problems with LLMs.
4 Experiments
4.1 Experimental Setup
Models We use Gemini 1.5 Flash our primary VLM, specifically the Gemini- 1.5-flash-
checkpoint via the Vertex API [18]. This is because Gemini is already the state-of-the-an on a number of video question-answering (QA) datasets (Tab. and it supports long context lengths for fair comparison of baseline inference to our method. Gemini uses 258 tokens per frame, and unless otherwise specified,
a context budget of 32K tokens. This corresponds to 120 frames leaving sufficient remaining tokens for the input question. To show the generalisability of our method, we also use Qwen-2.5-VL [6] and GoT--o-mini
(I. The prompts for all models are detailed in App. D
For all datasets, we sample videos at I frame per second (fps) following [55] 58 61 72).
Datasets We use the following long-video QA datasets:
5

---

Egoschema 41 is popular benchmark derived from Ego4D [9]. It consists of 5-way multiple choice questions on videos which are 180 seconds long. We run ablations on the subset of 500 labelled examples, and also report results on the full set of 5000 videos via the evaluation server.
LVBench [58] is recent dataset with an average length of 4080 seconds (68 minutes), and four multiple choice options. Videos are from YouTube, and since some are no longer available online, we include our full list of video ids in App. D We use visual inputs only.
OpenEQA 40] is a recent dataset targeted at embodied QA for mobile agents. adds open-ended questions to the HM3D 477 and ScanNet [13] datasets, where videos on average 452 frames long. As the answers are open-ended, the standard protocol is to use an LLM, specifically GPT-4 [2]. to score the predicted answer against the ground truth answer using scale from I to 5. These results are then averaged and normalised to score out of 100.
NEXT-QA 666 is a popular video QA dataset with five multiple choice options. Videos are average only 39.5 seconds long. However, we report results on this dataset to compare to prior works.
Table I: Comparison of different context aggregation approaches. All methods take 120 frames as input, using a 32K wondow for Gemini Flash, Our improvements are larger on the longer LVBench dataset.
4.2 Ablation Studies
Context Aggregation Analysis Table compares different context aggregation functions, G, (Sec.3.2) on both Egoschema and LVBench. Baseline inference performs no context aggregation and answers the question directly. We use a 32K token con- text limit, corresponding to 120 frames.
Egoschema
LVBench
Baseline inference
72.6
50.3
Single-step Hierarchical Dynamic-segment
74.8
48.3
74.0
53.3
75.2
61.7
In addition to Single-Step and Synamic-Segment TCoT (Sec. 3.2 we consider an additional approach which can iteratively process long video, which we denote Hierarchical TCoT. As detailed in App. D we perform Single-Step TCoT iteratively: First we coarsely sample frames from the video, then once we have identified frames of interest, we sample nearby frames that were not initially considered and iterate our aggregation procedure until the selected context has not changed, or the maximum number of iterations have been reached.
On Egoschema, all context aggregation methods, including the single-step variant, improve over baseline inference. Egoschema videos are short (180 frames), and effectively fit into the model's context window. The fact that all approaches improve over the baseline emphasise the utility of curating the model's 's input context to remove distractors.
far longer (average of 68 minutes). our proposed Dynamic- Segment TCoT (with I = 12) shines. This approach is able to effectively consider the whole video whilst adhering its 32K context limit per VLM call. Single-step TCoT does not improve over the baseline here; we posit this is due to processing only a sparse subset (120 of average of 4080) of the total frames, which is too few to identify the relevant frames. Hierarchical TCoT mitigates this strategy, but it underperforms our partitioned aggregation as it
On LVBench, where videos
problem by adopting a coarse-to- may miss short events which are not present in the initial, coarse sampling of the video.
We used s = 64 frames, and = 12 segments for our experiment, ablating this choice App. A
Computation vs accuracy analysis We analyse the trade-off between computational cost and accuracy of TCoT in Fig. Jon LVBench. We consider two alternatives: First, we perform standard inference with Gemini using a larger context limit. And second, as an alternate inference-time scaling strategy, we use "self-consistenee take the majority vote as the final answer. These multiple results are obtained by randomly sampling 120 frames from the input, and also by increasing the sampling temperature to 0.7 [59] to obtain diverse output. We use the total number of tokens processed by the VLM to measure computation, as
[59], where we sample multiple predictions from the VLM and
it is directly proportional to the monetary cost of using the model via an API. Moreover, metrics such as GFL.Pss and inference-time are not available when calling the model through an API.
Figure shows that as we increase the number of segments, I, and therefore the total number of tokens/ / frames processed, the performance of TCoT increases smoothly, whereas standard baseline inference saturates at around 1000 frames (264K tokens). When processing total of 2700 frames (700K tokens), we are able to achieve an improvement of 2.8 points (61.7 77 vs 58.9) at the same cost.

---

As our approach curates the relevant context from the input video, and
62
Accuracy LVBench
60
LVBench videos are very long with an average of 4080 frames, TCoT is not as effective when the total num- ber of frames considered is low. This is because there are not sufficient in- put frames to select the most relevant observe
58
TCoT (Ours)
56
Baseline
54
Self consistency CoT
52
50
context from. Therefore, the benefits of our approach over the long-context baseline after processing a total of 512 frames (132K tokens).
31K
264K
528K
(1.20)
(512)
Computational Cost: Visual tokens and (Frames)
Figure 4: Accuracy vs computation trade-off. We compare Temporal Chain of Thought to two alternatives: baseline in- ference and self-consistency CoT [59]. We use the total num- ber of visual tokens and frames (in parentheses) processed to measure computation, and vary 7 from 2 to to do for TCoT. Our approach improves consistently whilst baseline inference saturates in the presence of distractors from more frames. Self-consistency CoT is ineffective, underlining the need for inference-time scaling approaches tailored video.
Self-consistency chain-of-thought prompting [59], achieves minimal improvements over just single inference call, highlighting that inference strategies developed for language are not directly applicable to video, which requires specialised approaches as ours.
s = 64 frames, TCoT does not exceed 32K input context tokens, regardless of / and the total number of tokens frames processed. Baseline inference, contrast, is restricted by the context window supported by the model.
Note that as
Alternate context aggregation approaches. Table 2 compares the following methods for choosing a fixed number of frames (120 to fit a 32K context limit) from a video:
Uniform sampling: We uniformly select 120 frames.
Feature similarit: We select frames based on their similarity the embedded question using k-nearest neighbour search, as explored by 16 62). We embed either the captions generated by our VLM ("question -> captions") or the frames directly ("question frames") using the strong dual-encoder SigLIP model [71]. For "question - captions". we prompt our VLM to generate concise captions to ensure they within the 64-token context limit of the SigLIP text encoder (7I). VI.M-Based: We use a VLM for selecting relevant context, either based on captions generated from the frames (as done by (2I), by directly feeding the frames without any intermediate captions. We use the same partitioned segments (Sec. 3.2) in all cases.
We observe that all context aggrega- tion approaches outperform naive uni- form sampling both datasets, high- lighting the importance of curating the input to VLM. Second, note that methods selecting frames directly with VLM (rows 4-6) perform bet- ter than those relying on feature sim- ilarity (rows 2-3). Intuitively, this is because instruction-tuned VL.Ms can adapt easily new task in shot manner, and also dynamically vary the number of selected frames de- pending on the question type as shown Fig. 6 Third, when selecting with VLM, directly selecting from frames
Table 2: Alternate context aggregation strategies. We compare different methods for selecting 120 input frames. We use the same VLM, with 32K-context for the final answer.
Selection strategy
Egoschema LVBench
Uniform sampling
72.6
50.3
Feature similarity
Question + captions
73.8
52.1
Question + frames
73.4
54.4
VIMM-Based
Select from "concise" captions
74.0
58.3
Select from "long" captions
72.8
60.4
Select directly from frames (Ours)
75.2
61.7
Oracle annotated time references
67.4
as in our method (row performs better than using intermediate frame captions which lose infor-
mation. Moreover, captioning based approaches are sensitive to the prompt used for captioning:
"concise" and "long" captions perform differently for Egoschema and LVBench (rows 4 and 5). Our captioning prompts are detailed in App- [} Finally, we observe that when we use the oracle time-reference frames which are numan-annotated for L.Bench [58) our performance increases by 5.7 points. The largest headroom, however, is in improving the answerer model (ie VLM) which is not the scope of this work.
7

---

927
920-477
1814
Queiter: Mani nouuo vouer poo cul spen where sru bo buyn bus prodecoi
Duayttor fion the mve vilguee wioo warn best Bernnot
Bauden anawrr To
bost (huppoane
pllte stow (ilc
snower to
erower
Manderr salected fooneslly rtall
Partal volleiittonn jnstittttinnnn
Mumber of rekdddd hamma by Touth-d humm
210- hame
and TIMETO crenaccusz
2000 10000
with
and
frems Farm cratonelly auraee
plaa
Romen
wer out can rason otter suarach achool, and the antaraaaaeeee medic go acook
Trame cots 1205, The thamn alow logaer nouuing but usin spoin to dil luddy
sthecito ti then shopping
anwerr judiiiatoon slgger actating chazuddda but shap taet sppiie
beauty products. shardork, la meaion gasoutage aagin anachadd.
777
Quaition:: the
country dan sqaiend arger
Guette What is now wher intses dlbe bodinn edie sheiter
Optinn jui 4a% Bi Saa g0-42
suther
arower Sher coers
ernwen
X
ing inportion ina crurtry Thorliie town ty sewwer sor X
enreer-
Falkur neched The whiigg
are sulettdd Durng there enver pore on Micamnr whech laly rection
the
Partal on tastittinns
rean
Partel VVM
Freme aoco van
her refhetion
inpessinn 49%, od catngghurceee chw.gttteeeeeeeeee afrecio sertingeeeen sereee andar thehe Jay inlorethe sarvey Mkraswweree jarkerton vide shous alar grigh hightst prootaag age being
Figure 5: Qualitative examples on LVBench. Note how model focuses different parts of the video to make its prediction. For clarity, we sample frames from the segments selected by TCoT. In the failure case, although TCoT finds various frames showing the fairy's 's reactions, none of them include the drop water, meaning that the answerer cannot make the correct prediction.
Porcntnge of solocted tromes (N)
Dynamic Context Aggregation Figure 6 analyses the percentage of selected frames across different question types on LVBench. Observe how the proportion of selected frames
%
TCcT
Annotated onformnces
adapts dynamically based on the
question type. For example, temporal grounding questions focus on specific moments in time, and accordingly less than 10% of the frames are chosen. Summarisation leads to the highest proportion of selected frames, which intuitively agrees with the need for a broad understanding of the video. with the proportion of human-annotated "time-reference" frames (58). Appendix TonddixA]shone shows that selected frames show distinct behaviours across datasets, confirming that our approach is indeed select a larger proportion of frames, which agrees with the fact that was annotated by [41] to require looking at at least 56% of the video. Finally, App. [A also analyses performance by question type, showing that TCoT shows the largest improvement over the baseline for "temporal grounding" and "key information retrieval" which requires precise localisation of relevant context.
Key Information Theiprrl
Entity
Foent understanding
Resseniing Summarientoon
groundting
reoignisinn
Figure 6: Percentage of selected frames by question type. Observe how the proportion of selected frames dynamically adjusts to the question type, aligning well with the human- annotated time-reference frames on LVBench (58).
Note that our selected proportions are also in agreement
adaptive. On Egoschema,
Qualitative examples
Figure and App. present both success and failure cases of our approach.
4.3
State-of-the-AAt Comparison
Finally, compare the state-of-the-art on four video QA datasets in Tab. focusing long-video datasets with prior works leveraging VLMs.
LVBench We substantially outperform prior works this recent, challenging dataset (Tab. with an average length of 68 minutes. Note that our Gemini 1.5 Flash baseline that we used in our ablations (Sec. 4.2) was already state-of-the-art, and achieved substantial improvements: Namely, we improve by I1.4 points with the same 32K context limit, and by 2.8 points when processing the same number of total tokens (700K) with TCoT method.
To show the generalisability of TCoT to other VLMSs we also present results using Qwen 2.5 7B (6] and GPT-4o-mini [ We run baseline inference for both models with the maximum number

---

Table 3: State-of-the-art comparison. We report our own Gemini 1.5 Flash baseline for Egoschema and LVBench as it outperforms [55] [58. For LVBench and OpenEQA, we report the tokens used in a window, context-window, and the total number of tokens processed. OpenEQA uses the "LLM-as judjed protocol of using GPT-4 to evaluate the answer. t: Our reproduction.
single context-
LV LVBench
Model
Context
Total
Accuracy
Egoschema and NexI-QA
VidcoAgent
16
37.6
Eguscherna Subset
NeXT-QQA
Intem_VL22.5-788
12
43.6
Method
LLM/ VLM
Full set
Accuracy
Qwen-2.5.VL.
6
128K
128K
46.1
GPT-4o-mini
22K
LepoRepo 28
Mixtral
66.2
41.2
60.9
Gemini 1.5 Flash 555
32K
32K
50.3
MoreVQA 42
PaLM-2
51.7
69.2
Gemini 1.5 Flash
55
700K
700K
58.9
Video Agent 61
GPT-4
60.2
54.1
71.3
TCoT (Qwen-2.5-VL-7B)
128K
320K
49.1
Video Agent 61
Gemini 1.5 Flash
65.6
TCoT (GPT-4o-mini)
22K
86K
53,5
LLoVi 72
GPT-4
61.2
$2.2
73,8
TCuT (Gemini 1.5 Flash)
32K
672K
61.7
GPT-4V R
GPT-4V
63.5
55.6
LVNet 45
GPT-40
68.2
61.1
72.9
MotionEpic 77
Vicuna
76.0
Open EQA
Video'rree [2
GPT-4
66.2
61.1
75.6
Model
Context
Total
LLM Match
BOLT X8
LLaVA-One
60.6
64.0
79.5
Claude
21 CLU C00
6K
6K
36.3
LongVU
Qwen2-7B
67.6
GPT-4V
4.2K
4.2K
55,3
Gemini Flush
55
Gemini 1.5 Flash
72.9
67.8
80.0
Gemini 1.5 Flash
77.4K
77.4K
68.0
TCoT (ours)
Gemini 1.5 Flash
75.2
69.1
81.0
TCoT (Gemini 1.5 Flash)
32K
76.4K
69.2
of frames/ / tokens that it supports to obtain the strongest possible baseline. Namely, this is 1024
frames 128K for Qwen 2.5 and 250 frames / 22K for GPT-4o-mini (the GPT API supports only 250 frames, even though it supports more text tokens in its context (]. Our improvements with TCoT are consistent and considerable on these VL.Ms, especially since our iterative TCoT allows us to consider more frames than the model supports natively in its context window. In particular, our improvement over GPT-4o-mini is the largest, at 5.5 points, as it supports the smallest context window.
Egoschema Table shoow that we outperform prior works on Egoschema. We compare to prior works that used either VLMs or LLMs (in conjunction with per-frame captioners) [45 61 62 72). We also re-implement VideoAgent [6]] (details in App. D using the same Gemini Flash VLM. Methods which initially compute captions on each frame are bounded by the quality of these captions, which often miss details relevant to the question. The Gemini 1.5 Flash baseline that we used in our ablations (Sec. (4.2) was already state-of-the-are and we improve upon further with TCoT.
NEXT-QA The videos in NExT-QA are the shortest from all of our evaluation datasets, and average only 39.5 seconds. Nevertheless, we this dataset to compare to prior works this dataset which have also leveraged VLMs or LLMs (operating on per-fram ter-frame captions) in Tab. 3 There is less headroom for us to improve on this dataset, as the videos are short and the accuracy of our Gemini 1.5 Flash baseline is already high. Our results are still consistent with our other experiments and Egoschema. Namely, we outperform prior works and we reduce the relative error of our state-of-the- art Gemini 1.5 Flash baseline by 5.0%.
OpenEQA OpenEQQ presents different domain, as the questions- answer pairs were labelled for mobile, embodied agents. Moreover, the answers are open-ended and an LLM (GPT-4 22), is used to evaluate answers in the authors' protocol [40]. Consistent with other datasets, we outperform prior works and our strong Gemini 1.5 Flash baseline using 300 frames (77.4K tokens).
5 Conclusion
We presented Temporal Chain of Thought, an inference strategy for in VLMs, motivated by the fact that VLMs are affected by redundant information in their input context (3 28 37]. Our approach curates the input context VLM by decomposing a video QA task into first adaptively finding the relevant frames in the video. We demonstrated the efficacy of this approach by achieving state-of-the-art results on different datasets. particular, our approach with an input context of 32K tokens outperformed 700K token model by 2.8 points on the challenging LVBench datasets where videos are 68 minutes long on average.
answering
Limitations Our approach, which we have shown to be effective across three different VLMs, nevertheless requires model with good
capabilities in order to perform the selection function (Fig. (3) in zero-shot manner. And whilst we have shown significant improvements from our inference strategy, future work could improve this further by training the model explicitly for this inference approach via reinforcement learning 20 44).

---

References
[1] Open AL Gpt-4o mini.
[2] Open AL Gpt-4 technical report. In arXiv preprini www.::23033000011101 2023.
[3] Open AI. Gpt-4v(ision) system card. 2024.
Anthropic. The claude model family: Opus, sonnet, haiku. 2024.
[5] Kirolos Ataallah, Xiaoqian Shen, Eslam AbdeIrahman, Essam Sleiman, Mingehen Zhuge, Jian Ding. Deyao Zhu, Jürgen Schmidhuber, and Mohamed Elhoseiny. Goldfish: Vision-langual understanding of arbitrarily long videos. ECCV, 2024.
[6] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, et al. Qwen2. 5-v1 technical report. arXiv preprint arXiv:2502. 13923, 2025.
[7] Ivana Balaževié, Yuge Shi, Pinelopi Papalampidi, Rahma Chaabouni, Skanda Koppula, and Olivier J Hénaff. Memory consolidation enables contecontext video understanding. In ICML, 2024.
[8] Daniel Bolya, Cheng-Yane Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. ICLR, 2023.
[9] Bradley Brown, Jordan Juravsky. Ryan Ehrlich, Ronald Clark, Quoe V Le, Christopher Ré, and Azalia Mirhoseini. Large language monkeys: Scaling inference compute with repeated sampling. In arXiv preprint or 21v 2024.
[10] Shyamal Buch, Cristóbal Eyzaguirre, Adrien Gaidon, Jiajun Wu, Li Fei-Fei, and Juan Carlos Niebles. Revisiting the" video" video-language understanding. In CVPR, 2022.
[11] Shyamal Buch, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid. Flexible frame selection for efficient video reasoning. CVPR, 2025.
[12] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries open-source multimodal models with model, data, and test-time scaling. In arXiv preprini arXiv:2412.05271, 2024.
[13] Angela Dai, Angel Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias NieBner. Scannet: Richly-annotate 3d reconstructions of indoor scenes. In CVPR, 2017.
[14] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang. Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general -prepoe Iisounlaagguaee models with instruction tuning. In NewrlPS, 2023.
[15] Alexander Dunn, John Dagdelen, Nicholas Walker, Sanghoon Lee, Andrew Rosen, Gerbrand Ceder, Kristin Persson, and Anubhav Jain. Structured information extraction from complex scientific text with fine-tuned large language models. In arXiv preprint 21X.::::22220223:5 2022.
[16] Yue Fan, Xiaojian Ma. Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory- augmented multimodal agent for video understanding. In ECCV, 2024.
[17] Hao Fei, Shengqiong Wu, Wei Ji, Hanwang Zhang, Mcishan Zhang. Mong-Li Lee, and Wynne Hsu. Video-of-thought Step-by-step video reasoning from perception to cognition. In ICML, 2024.
[18] Google. Vertex api. https://cloud. google. wwwp::/ggggggoooooooooooeeeeoeee com/vertex-ai.
[19] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, al. Ego4d: Around the world 3,000 hours of egocentric video. In CVPR, 2022.
[20] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in lims via reinforcement learning. In arXiv preprint arXiv:2501. 10t::299...1988,, 2025.
[21] Songhao Han, Wei Huang, Hairong Shi, Le Zhuo, Xiu Su. Shifeng Zhang, Xu Zhou, Xiaojuan Qi. Yue Liao, and Si Liu. Videoespresso: A large-scale chain-of-thouge thtouugh dataset for fine-grained video reasoning via core frame selection. arXiv preprint arXiv:241 14794, 2024.
[22] Wenyi Hong. Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang. Zihan Wang. Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents, In CVPR, 2024.
10

---

[23] Cheng- Ping Hsieh, Simeng Sun, Samuel Kriman, Shantanu Acharya, Dima Rekesh, Fei Jia, Yang Zhang. and Boris Ginsburg. Ruler: What's the real context size of your long-context language models? In arXiv preprint wrXiv:2404.06654 2024.
[24] Gautier Izacard, Mathilde Caron. Lucas Hosseini, Sebastian Riedel, Piotr Bojanowski, Armand Joulin, and Edouard Grave. Unsupervised dense information retrieval with contrastive learning. TMI.R, 2022.
[25] Andrew Jaegle, Sebastian Borgeaud, Jean- Saan-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver I0: A general architecture for structured inputs & oulputs. In ICLR, 2022.
[26] Jingwei Ji. Ranjay Krishna, Li Fei-Fei. and Juan Carlos Niebles. Action genome: Actions as compositions of spatio-temporal scene graphs. In CVPR, 2020.
[27] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transac- tions on Big Data, 7(3):535 547, 2019.
[28] Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S Ryoo. Language repository for long video understanding. arXiv preprint arXiv:2403. /4622, 2024.
[29] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In NeurlPS, 2022.
[30] Bruno Korbar, Yongqin Xian, Alessio Tonioni, Andrew Zisserman, and Federico Tombari. Text-conditioned resampler for long form video understanding. In ECCV, 2024.
[31] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava- next-interleave: Tackling multi-image, video, and 3d in large multimodal models. In arXiv preprint arXiv:2407.07895, 2024.
[32] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Rootstrapping language-image pre-training with frozen image encoders and large language models. JCML, 2023.
[33] Yicong Li, Xiang Wang. Junbin Xiao, Wei Ji, and Tat-Seng Chua. Invariant grounding for video question answering. In CVPR, 2022.
[34] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. In NeurlPS, 2023.
[35] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lec. Llava-next: Improved reasoning, ocr, and world knowledge, 2024.
[36] Hao Liu, Matei Zaharia, and Pieter Abheel. Ring attention with blockwise transformers for near-infinite context. In ICLR. 2024.
[37] Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang, Lost the middle: How language models use long contexts. Transactions the Association for Computational Linguistics, 12:157 173, 2024.
[38] Shuming Liu, Chen Zhao, Tianqi Xu, and Bernard Ghanem. Bolt: Boost large vision-language model without training for long-form video understanding. In arXiv preprint 1rXiv:2503.2148. 2025.
[39] Yongdong Luo, Xiawu Zheng, Xiao Yang. Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually-aligned aetrieval-uugmented long video comprehension. In arXiv preprint arXiv:2411. 13093, 2024.
[40] Arjun Majumdar, Anurag Ajay. Xiaohan Zhang, Pranav Putta, Sriram Yenamandra, Mikael Henaff, Sneha Silwal, Paul Mevay, Oleksandr Maksymets, Sergio Amnud, al. Openeqa: Embodicd question answering in the era of foundation models. In CVPR, 2024.
[41] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very lorg-form video language understanding. In NeurlPS, 2023.
[42] Juhong Min, Shyamal Buch, Arsha Nagrani, Minsu Cho, and Cordelia Schmid. Morevqa: Exploring modular reasoning models for video question answering. In CVPR, 2024.
[43] Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, et al. Pivot: Iterative visual prompting elicits actionable knowledge for vlms. In arXiv preprint arXiv: 07872, 2024.
11

---

[44] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, al. Training language models to follow instructions with human feedback. In NeurIPS, 2022.
[45] Jongwoo Park. Kanchana Ranasinghe, Kumara Kahatapitiya. Wonjeong Ryoo, Donghyun Kim, and Michael S Ryoo. Too many frames, not all useful: Efficient strategies for long-form video qa. In arXiv preprint or Xiv:24006 09396, 2024.
[46] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In ICML, 2023.
[47] Santhosh K Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alex Clegg, John Turner, Eric Undersander, Wojeiech Galuba, Andrew Westbury, Angel X Chang, et al. Habitat-matterpon 3d dataset (hm3d): 1000 large-scale 3d environments for embodied ai. In arXiv preprint arXiv:2009008238, 2021.
[48] Michael S Ryon, AJ Piergiovanni, Anurag Amab, Mostafa Dehghani, and Anelia Angelova. Tokenlcarner: What can 8 learned tokens do for images and videos? In NeurlPS, 2021.
[49] Xiaoqian Shen, Yunyang Xiong. Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu; Spatiotemporal adaptive compression for long video-language understanding. In arXiv preprint arXiv:2410.17434, 2024.
[50] Yunhang Shen, Chaoyou Fu, Peixian Chen, Mengdan Zhang, Ke Li, Xing Sun, Yunsheng Wu, Shaohui Lin, and Rongrong Ji. Aligning and prompting everything all once for universal visual perception. In CVPR, 2024.
[51] Noah Shian, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. In NeurlPS, 2023.
[52] Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.03314, 2024.
[53] Quan Sun, Jinsheng Wang. Qiying Yu, Yufeng Cui, Fan Zhang. Xiaosong Zhang, and Xinlong Wang. Eva-clip-18b: Scaling clip to 18 billion parameters. In arXiv preprint vxXiv:2402.04255 2024.
[54] Dídac Surís. Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In ICCV, 2023.
[55] Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. In arXiv preprint arXiv:2403.0533 2024.
[56] Peng Wang, Shuai Bai, Sinan Tan, Shijic Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world any resolution. In arxiv preprint arXiv:2409.12/99 2024.
[57] Shijie Wang, Qi Zhao, Minh Quan Do Nakul Agarwal, Kwonjoon Lee, and Chen Sun. Vamos: Versatile action models for video understanding. ECCV, 2024.
[58] Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Shiyu Huang, Bin Xu, Yuxiao Dong, Ming Ding, al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08033, 2024.
[59] Xuezhi Wang. Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. arXiv preprint arXiv:2203.11171) 2022.
[60] Xijun Wang, Junbang Liang. Chun-Kat Wang. Kenan Deng. Yu (Michael) Lou, Ming Lin, and Shan Yang. Vila: Efficient video-language alignment for video question answering. In ECCV, 2024.
[61] Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video under- standing with large language model as agent. In arXiv preprint arXiv:2403. 10517, 2024.
[62] Ziyang Wang, Shoubia Yu, Elias Stengel-Eskie Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for Ilm reasoning on long videos. In arXiv preprint 2405: 19209, 2024.
[63] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. ofthught prompting elicits reasoning in large language models, In NewrIPS, 2022.
12

---

[64] Tsung-Har Han Wu, Giscard Biamby, Jerome Quenum, Ritwik Gupta, Joseph E Gonzalez, Trevor Darrell, and David M Chan. Visual haystacks: Answering harder questions about of images. In arXiv preprint urXiv:2407.33760 13766, 2024.
[65] Yangzhen Wu. Zhiqing Sun, Shanda Li, Sean Welleck, and Yiming Yang. Inference scaling laws: An empirical analysis compute-opine optimal inference for problem- with language models. arXiv preprint XxXv:2448.00074, 2024.
[66] Junbin Xiao, Xindi Shang, Angela Yao, and Tar- Tat-Seng Chua. Next-qa: Next phase question answering to explaining temporal actions. In CVPR, 2021.
[67] Jiaqi Xu, Cuiling Lan, Wenxuan Xie, Xuejin Chen, and Yan Lu. Retrieval-based video language model for efficient long video question answering. In arXiv preprint arXiv:2312.04931 2023.
[68] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In NewrIPS, 2023.
[69] Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model video localization and question answering. In NewrIPS, 2024.
[70] Tao Yuan, Xuefei Ning, Dong Zhou, Zhijie Yang. Shiyao Li. Minghui Zhuang, Zheyue Tan. Zhuyu Yao, Dahua Lin, Boxun Li, et al. Lv-eval: A balanced long-context benchmark with 5 length levels up to 256k. In arXiv preprini 2rXiv:2402.0511 2024.
[71] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023.
[72] Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple IIm framework for long-range video question-answering. In EMNLP, 2024.
[73] Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In CVPR. 2023.
[74] Zijia Zhao, Haoyu Lu, Yugi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, and Jing Liu. Needle in video haystack: A scalable synthetic framework for benchmarking video milms. In arXiv preprint arXiv:2466. 09367, 2024.
[75] Xingyi Zhou, Anurag Arnb, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. In CVPR, 2024.
13

---

Broader Impact
Our work presents an inference-strategy improve the performance of Vision Language Models (VLMs) on long videos (our improvements are the largest on hour-long videos), VLMs, and video guestion-answeringg the domain of our work, is a generic technology with a wide range of potential applications. We are unaware of all potential applications, but we are cognizant that each use-case has its own societal impacts depending on the intentions and motivations of the individuals or organisations building and deploying the system. For example, long-video understanding can be used for productive purposes, such as a user asking system detailed questions about long, instructional videos. However, it may also be used for applications such as surveillance systems.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the paper are made at the end of the introduction section. Each of these claims are verified experimentally in the experimental section (Sec. 4. The
novel VLM inference strategy that proposed is contrasted to prior works in Sec. 2 and detailed in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the
contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitations of our work are described in the Conclusion section (Sec. 5)
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
14

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: There are no theoretical results in this work.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the main information required to reproduce experimental results Exhaustive experimental details are included in App. [ particular, we have included all VLM prompts used in experiments. We have used VLMs which can be publicly accessed via APIs or open-source weights, and have used publicly-available, academic datasets for evaluation. For LVBench (58) which is a YouTube-basee dataset, we have included a list all Video IDs since videos can be removed from the platform. Guidelines:
in Sec. 4..T
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
15

---

(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily
model architecture, the paper should descrihe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing verifying the results. 5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We have used publicly-availall academic datasets (Egoschema, Next-QA, LVBench and Open-FQA) for all of our experiments. To the best of our knowledge, the paper contains sufficient details to faithfully reproduce the experimental results. We are not, however, releasing our experimental code. Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.ca
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: for more details.
/ //nips. cttpccccpppccccccccccccccccccccc gww.eeidddeeedubmmmiiimm
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
reproducible, they
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All experimental settings and details are described in Sec. 4. and App. D
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
16

---

7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Our experimental results in the ablation study (Sec. 4.2) are averaged over 3 runs, control for random variability. And we therefore believe that our results and conclusions are signficant. We have not run further variations of our experiments as it is too computationally expensive do so, as we are using VLMs for each experiment.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
This described in App. D We call the Gemini 1.5 Flash 55 and GPT-4o- mini II through their public APIs. Qwen-2.5-VL. [6] has publicly-available weights, and we run the HuggingFace implementation using a server with 8x NVIDIA A100 GPUs.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https::///trrippss.s http://ss.pppp.ooeoccccccssseeea
Answer: [Yes]
Justification: All authors have read the NeurlPS Code Ethics, and attest that this research conforms in every aspect with it.
17

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We include Broader Impact statement as the first section of the appendix, just before this checklist.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper does not release any data or models.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
18

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have cited the original authors or all models and datasets used in the paper. We have complied with all available licenses for both models and datasets. We have used only publicly-availabe academic datascts, for only academic purposes.
We include further information of each asset below:
Next-QA [66
License: MIT
LVBench
58
License: CC BY-SA 4.0
OpenEQA
License: MIT
Egoschema
License: Unknown.
Qwen-2.5-VI VL
License: Apache 2.0
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperr thcode. thccode com cwm/catoatac/o datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
19

---

for The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: Our paper describes novel inference strategy for VLMs for video data. Therefore, our paper describes our method, relation to prior work and experimental results with our approach in detail.
However, we did not use an LLM at to write the paper, or to write the code that we used to conduct experiments.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy what should or should not be described.
20

---

Table 4: Independent segment answer aggregation: As an additional baseline to show that our method can reason across different segments, we partition videos into segments, answer questions independently within each window, and aggregate these answers to predict the final response. This baseline substantially underperforms our final method (Sec. 3.2) as it cannot reason across different segments.
Model variant
LVBench
Individual segment answers
51.0
Individual segment answers with high confidence prompt 55.6
Ours
61.7
LVBench
25
Egoschema
20
Percentage
15-
10
5-
0
20
40
60
80
100
Initial proportion of frames selected (%)
Figure 7: Distribution of frames selected by our model. Our model selects the relevant context in an adaptive manner, choosing a greater proportion of input frames for Egoschema (mean 74%) than for LVBench (mean of 15%) The results on Egoschema correlate with its "temporal certificate" 41
Appendix
In this appendix, we include additional experimental results (App. A), detailed failure mode anal- ysis (B. qualitative examples (App.C), Cp. well additional experimental details (App. D). References follow the numbering from the original paper.
A Additional experimental results
Independent segment answer aggregation To show that our approach is able to reason across different segments in the video, we perform the following experiment with alternate baseline: We divide videos from LVBench of N frames into [ segments of length s, and answer the question independently with justification within each window. To compute the final answer, pass each of the per-segment answers again to the VLM, and ask predict the final answer. In Tab. 4 we observe that the performance is substantially lower LVBench: 55.6% compared to 61.7%. Intuitively, the final call to the VLM is often noisy because most segments are irrelevant. We found that the model should only propose an answer for a segment if it is highly confident ("high confidence prompt" in Tab, 4 otherwise, it attempts to answer for every segment, leading to contradictory and noisy information the final VLM call. Additionally, this baseline underperforms compared to our method (Sec. 3.2 because many questions necessitate considering multiple segments. Therefore, our approach not only decomposes video into smaller segments but also reasons across these segments, a capability lacking in simpler baselines.
Distribution of selected frames. To further analyse the adaptivity of our method in selecting frames relevant to the question, Fig. 7 Janalyses the proportion of frames selected. Concretely, we plot the ratio of frames initially selected by TCoT (Eq. 5). We note that if the number of selected frames is larger than the context-limit of the model, k. reduce by uniformly sampling within the sorted list frame indices (Sec. 3.2). Nevertheless, it is more informative to analyse this initial distribution, it shows the frames which the model deems relevant.
Our selection function shows distinct behaviour across Egoschema and LV Bench, confirming that is indeed adaptive to the input questions. The proportion of selected frames is significantly higher on
21

---

Basellne
Annolated time references
TCoT (Curs)
Average eocuresy
80
40
Information
Temporal grounding
Entity
Event underslanding
Summarisation
recognition
Figure Performance per question type on LVBench. We compare baseline inference, our TCoT method, and using the oracle of the numan-annotate time references to select relevant frames. We achieve significant improvements on most question types, and often near the accuracy of the oracle, particularly on "key information retrieval" and "temporal grounding" which requires precisely locating the relevant information the long video.
Table Limits of Chain of Thought (CoT) methods in long video understanding. Existing CoT techniques developed for language do not bring significant improvements on long video understanding. This calls for CoT techniques tailored to this task.
Chain of Thought (CoT) technique
# VLM calls
LVBench
None Zero-shot CoT
1 49.5
1
50.3
Zero-shot CoT with 2-stage prompting 29
2
49.4
Zero-shot CoT with self-consistency [59]
9
51.7
Egoschema, its questions were designed by [1 to require a more holistic understanding of the video. Indeed the "temporal certificate" [4]] (the duration of the video that an annotator must look at to answer the question) is estimated to be at least 100 seconds, or 56% of the video, which correlates with Fig. Although LVBench videos are longer, their questions require localising specific moments in the video, as also shown in Fig. 7
Performance per question type on LVBench. Figure Figure@compares compares the performance of our method to baseline inference, and the oracle of using the annotated time-reference frames to select relevant frames. We observe that TCoT consistently outperforms baseline inference, particularly for "key information retrieval" and "temporal grounding" which requires precise localisation of relevant context. Moreover, we approach the accuracy of the oracle on these categories too. However, we note that baseline inference performs better for summarisation questions, which require holistic view of the video. This is also the only question type in Fig. 6 where our method on average selects fewer frames than the oracle, suggesting that we do not select enough relevant information in these cases.
Limitations of pure text CoT on long video reasoning. Chain of Thought (CoT) prompting techniques enhance the reasoning capabilities of L.MMs across various tasks. Inspired by these successes, we explore whether typical CoT methods can also improve performance in long video In partieular, investigate:
understanding in Tab. 5
Zero- CoT 29
We add a sentence encouraging step-by-step thinking, namely "Explain your reasoning," into the prompt before outputting the final answer (Fig. 14p.
Zero-Shot CoT with two-stage prompting 29
we append the output of the initial Zero-shot CoT to the prompt and generate new prediction.
Zero-Shot CoT with self-consistency a different set of frames at each run to encourage diverse reasoning across runs. The final prediction is determined by majority voting from nine runs, each employing Zero-Shot CoT. We experimented with more runs but observed no performance improvement (shown in Fig.
(59 we set the VLM temperature to 0.7 and sample
22

---

100
Accuracy
60
40
0
0
10 20
30
40
50
60
70
80
90
100
Initial proportion of frames selected (%)
Figure 9: Accuracy according to the proportion of frames selected on Egoschema. Observe how model's accuracy remains consistent, regardless of the number of frames selected, suggesting that our TCoT method can effectively and adaptively select the relevant frames to answer the question. The dashed line shows the overall average accuracy on the dataset.
Table 6: Effect of hyperparameters. We analyse the effect of the segment size, 8 (a), and the number of uniform context frames, u (b). The context-limit is = 120, meaning that the remaining frames are selected by the model.
m Effect of segment size,
(b) Effect of uniform context, u.
Egoschema
LVBench
m
Egoschema
LVBench
73.0
56.8
75.2
57.8
16
73.6
57.0
88
32
74.6
58.5
32
73.8
58.1
64
56
73.2
59.3
64
75.2
57.8
32
74.0
56.2
120
73.8
57.8
120
72.6
50.3
observe that all the pure linguistic CoT techniques result in only marginal performance improvement on LVBench. This result calls for VLM inference strategies specifically tailored to video understanding tasks such ours. Finally, note that we adopt "Zero-Shot CoT" as default prompting technique for the VLM call generating the final answer (Fig. 14
In Tab. 5
Accuracy is consistent across selected frames To further analyse the quality of our frame selection, Fig. compares the accuracy of our TCoT model as function of the proportion of frames initially selected.
a model is effective in adaptively selecting only the frames that it needs to answer the question, then its accuracy should remain constant regardless of the number of frames selected. Figure 9 shows that this largely the case for our TCoT model, highlighting that our approach effectively selects the frames it requires for the task.
Effect of hyperparameters, and a Table shows the effect TCoT hyperparameters (Sec. 3.2 (segment size) and (uniform context).
Table 6a shows that the performance our model is relatively consistent across different segment sizes, on both Egoschema and L.Bench datasets. A smaller segment size means that require a smaller context window during context aggregation, whilst also requiring more VLM calls. Note that the total context-limit to the answerer, remains constant at k 120 here, enabling accurate predictions even with small window, s.
Table 6b varies the amount of uniform context, u, that we add. Once again, the total context-limin remains fixed at & 120, meaning that the remaining m
u frames are selected by the model. Moderate amounts of uniform context help on LVBench, as they enable the model to obtain broader awareness the video. Egoschema, contrast, does not benefit from uniform context, and this may be explained by Fig. 7 which shows that the model is already selecting a larger number of frames on this dataset, which has a "temporal certificate"
41 which covers the majority of the video. Finally, note that when 0, we effectively performing the standard inference baseline as frames
23

---

1.0
0.8
low recall & high precision
Precision
0.6
0.4
low precision & high recall
0.2
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Recall
Figure 10: Failure modes of TCoT We show the precision and recall of frames that have been
aggregated by our method, compared to the human-annotated time-reference frames LVBench. This is performed for the instances in the dataset where TCoT fails, while the oracle which uses only the rife-reference frames for answering succeeds.
being selected by the model. Overall accuracy degrades, particularly on LV L.Bench, as discussed in Tab.
B Failure mode analysis
We examine the failure modes of our Temporal Chain of Thought in detail in Fig. 10
To do so, we use the annotated "time reference" segments from LVBench [58), which are segments of video that contain the necessary information to answer the question accurately. Ideally, assuming that the time-reference annotations are completely correct, we should only select the time-reference frames and other ones.
In Fig. 10 we evaluate the precision and recall of our selected frames, where a Frame ID is denoted "true positive" if it falls within the annotated time reference segment and "false positive" if it does not. High precision means that few selected frames fall outside the time reference frames, while high recall indicates that most of the time reference frames included the selection. In Fig. IO we plot only the 168 instances where TCoT provides incorrect answers to the question, while the oracle answers correctly.
We identify prevalent failure modes in Fig. 10 and show qualitative examples for these modes in Fig.
II
Low precision and high recall (purple area in Fig. 10 qualitative examples in Fig. la The model selects too many frames, selecting frames that are only remotely related to the question. This over-eagerness leads to excessive and irrelevant frame selection. For example, we see in Fig. Tla (left) that if the question asks what the dragon is dropping while in the sky, the method selects frames of the dragon even when is clearly not the sky. Our model sometimes struggles with questions that require understanding connections between segments, such as determining the second-to-las event video as shown in Fig. T1a (right).

---

235
1357
None
30
128
Gssetlon: Waen piice sedpr saakpayy Pi oargen Binging the Mile
uhcaping Ootore
Howar
Guestoer aecootke sear, tesperamt mdees
mechanice trushe
ancwer::
Corrent armeeer: lhe mheche as loco stac tuap
cimpossbbe idently the "eecond bot cnar: to soure couneingu reguussre uargle vindow. Coroesurrty the necld aducta than tha ouch windon
uindhas
Thair io overt aokecorrily by the peetin woding to inscouncle
Partie VUM selection and Ristioatooon
Parial HLM celection and peciffiattioss
Prame ahowaa ocr dhving the fustors ofiic ausor Da laa sna votoo through Theae framss group mecharicaa meeting AT sncord
thagon helding flowrr
alooss
Frames Move the prean Brggen the prince and prinoess Mlliin-T
(a) Failures due to low precision
758-761
775-117
Questter: What
Quesiion: Wiat
fins
emower throwa emer
Our snawen vary aa
Falke our nethod fornes deo thmm can oelccied
Foluur reathad framee showing want The kook drop micoor new
Taee relermne reveenee el from 1i0e thanee 71 to Trome
Time referenoo from in Promells
phone 599)
frumon meli ien slow the worean dunciing withurd other people tefoor the soone
Prome sPowes resotion ofter unding reflection
Frame 1007 shows the worann trrwwing stoe.
Prome 1250 shanes reacionn expressir happinpss onios
(b) Failures due to low recall
Figure 11: Qualitative analysis of the failure modes of Temporal Chain of Thought. Failures due to low precision: Sometimes, our method selects too many frames. In the top left example, it is overly influenced by the question and selects frames at each window, even when they highly relevant. For instance, if the question asks what the dragon is dropping while in the sky, the method retrieves frames of the dragon it is. In the top right example. it is impossible to determine the "second last event" of the video by looking single window. As result, the model incorrectly tries to answer the question within each single window. Failures due to low recall: Sometimes, our method misses important frames, such as those showing "throwing the Coca-Cola can" (left) or "looking at drop of water as a mirror" (right). These missing frames are essential for accurate final inference.
when it is clearly not in the sky, incorrectly assuming
Low recall and high precision (orange area in Fig. and qualitative examples in Fig. lib): The selected frames are relevant, but the model misses crucial parts of the question. For example, in Fig. TTb (right) in response to the question, "What is the fairy's 's reaction when she first sees the doll after looking at a drop of water as a mirror?", TCoT selects frames of the fairy's reaction but omits frames showing the fairy looking the water drop. Consequently, the final answer is inaccurate because the final VLM call cannot determine which of the fairy reactions it sees occurs after looking drop of water.
C Qualitative results
Figure 12 visualises additional examples of TCoT method LVBench (58) following the same format as Fig. in the main paper.
Figure 13 shows example of why adding uniform context (described in Sec. 3.2 of the main paper) beneficial.
D Additional experimental details
Complete prompts Figure 3 in main paper showed our VLM selection call, (Eq. 4). For
completeness, we also include the prompt for answering. H, (Eq. 3 of the main paper) too. Specifically,
Fig. 14 and T5 shows the prompt for multiple-choice questions (Egoschema [1), LVBench (58) and
NeXT-QA QA [66] datasets), and Fig. I6 for open-ended questions (for the OpenEQA [4)] dataset).
25

---

1087
1137-1145
Time
Question:
meilice?
from
boxt (3) he pours the strust
Buoe the anawer: He pours Ros into the and udias 2.K
Our VLM serection justif cattons:
viooper
mas
pourin noo from poite.
Fe 121 frmme show algger weetin using appin dtrectty
Frame ID
1575-1578
1580-1583
2660
Time
effers
vloger sharcs the roft oilhe ofher guesss
Base the anewer: en The viogger hee los befoover rood X
ovoogger
with-ctin guosod
Our VLse sxtection Ionetonnnaae juatifiattoos catione:
natke inflating rettgenstor hand und paets
200 poo The hurn refrigeatoo full hooditamm
Our answorer justificutionn The iddo wiff tiftgge ge vib eoggers hoilel natigestoor for off vaaaous food tomn. "here intoorndioote indrouing that not" perator for shardd among gaints
Frame ID
9-13
2075
Time
Question: What the highest sercentage of the untty'y' respondents who oxparianced anger day provicus the survey (AI 48% 92% (D) 40% Base anewer: The video does not conten any informm metoon about survey on anger reperieced sed repponder in country. Therefore, (heue answerr
Our srawer:
VLM sxtection justin catione:
nerror mameratho bar grepn Worlca Anghher Countree"" The hightst the graphsh rapresero merro indketing repordarto eqerieood argar betore Ehesorayy
Figure 12: Additional qualitative examples LVBench. Note how our model focusses different parts of the video to predict the correct answer. The top row shows an example of the model focussing on 3 diverse segments the same video. The second row includes two such segments. The third row shows an example of how our TCoT approach is able to find the frames in the entire video of 2075 frames which contain the correct answer. all cases, for clarity, we show a single frame from each selected segment of frames. Faces have been blurred in the first row.
Frame ID
103-115
120
Question: offich foor westing machins?
appears
(Soore XX
Our enavert Blaad on provided video tramns the waahing machina i located on the second Roor. The video shoors hallvnrr on the second Roor with docra load bed rooma and lrandry room
mom
contain waiting machins, Bicore
frame selectiinn S1105
Our VLM selection justifications:
frumnes vallwae Perupper homoss
(0.
clunty wwwhing machine with focing www.enatiiiiiioooooo. aoy conaec shows bundyy
Figure 13: Importance of adding uniform context Our TCoT approach is able to select the relevant frames (103 to 115) for the question, by focussing on the washing machine (and provides the correct justification for doing so too). However, if pass only these selected frames to the answerer, the result incorrect as relevant information about which floor machine on lost. To alleviate this issue, we therefore include uniformly sampled context as well, visualised in this figure, and described in Sec. 3.2 of the paper. Example from OpenEQA 440]
26

---

You will be given question about video [number choices) possible answer options. You are provided frames from the video, retrieved by an intelligent agent.
Frames: (frame1), .... {frame N}
Question: {question]
Possible answer choices: {answer choices)
After explaining your reasoning, output the final answer in the format. "Final Answer: (X)" where X is the correct digit choice. Never say "unknown" or "unsure", "None", instead provide your most likely guess.
Figure 14: Multiple choice question prompt for the Gemini and GPT-4o-mini answering call, H
(Eq. 3).
Frames: (frame1),
[frame N}
Carefully watch the video and pay attention the cause and sequence of events, detail and movement objects and the action pose of persons. Based on your observations, select the best option that accurately addresses the question. Question: (question)
Options: (answer choices)
Answer with the option's letter from the given choices directly and only give the best option.
Figure 15: Multiple choice question prompt for the Qwen 2.5-VL answering call, (Eq. 3).
Hierarchical TCoT We include further details of hierarchical aggregation, which was introduced in Sec. 4.2 of the main paper.
We hierarchically extend Single-Step TCoT (Sec. 3.2 by iteratively sampling around the previously identified frames of interest. Intuitively, in long video where our model's context limit is far smaller than the number of input frames, if we first coarsely sample frames, we may miss many necessary frames. Therefore, once we have identified frames relevant to the question, we select nearby ones that were not considered initially and iterate.
Concretely, the first iteration follows "Single-Step TCoT" (Sec. 3.2 Given a set of frames, Xy sampled uniformly from the video, identify relevant frames, Xg, So S(x,, q). where Sg = (r,..... æ for j E Sn- Subsequently, we "zoom in" on these relevant regions by sampling additional frames around Xy (Fig. (p). Specifically, for each index in So, construct neighbour- hood of frames, X1 = f Ii...., Fitvi-.., -.. neighbourhood size, and duplicate frames are removed.
Fjtrh. where u denotes the
The new sequence frames, X1, serves as the input for the next iteration. We repeat this algorithm until convergence, namely until iterations have been completed earlier if the relevant set of frames has not changed, or other words, &,+1 &c.
Video Agent reimplementation We reimplemented the Video Agent framework [6] with several modifications to make it directly comparable to our approach. In particular, Video Agent used LaVila 73 as its captioner, whilst we use Gemini 1.5 Flash for this purpose for fair comparison. Similarly, instead of using EVA-CI.IP8B- -b-pus
[71] as the other baselines in Tab. 2 Finally, we used Gemini 1.5 Flash as the answerer to consistent with the rest of our work, instead of the original GPT-4. Video Agent starts off with initial number of uniformly selected frames. We ablated this hyperparameter and achieved the best performance with 50 initial frames Egoschema, and 64 initial frames LVBench.
53 we use
Frames: (frame1),
(frame N)
You will be given question about video. will be provided frames from the video, retrieved by an intelligent agent. It is crucial that you imagine the visual scene vividly possible enhance the accuracy of your response. Question: (question)
Figure 16: Open-ended question prompt for our Gemini answering call, H (Eq. 3).
27

---

Details of captioning for feature-similarity based aggregation An additional baseline that we used in the paper was retrieve frames based on the similarity to their captions (Tab, Here, we include additional details of it.
We generate types of captions for each frame of the video frame using Gemini 1.5 Flash. For concise captions, we use the prompt Write concise description of the image in sentence. For long captions, we the prompt Write a paragraph describing the image in detail. These captions were then embedded using the most performant version of SigLIP [7]]. SigLIP-So400m 14. We then used these embeddings to perform nearest- neighbour search, selecting the most similar captions the input questions. The frames corresponding to these captions were then used to aggregate the context, c, that was then passed to the answerer VLM.
Computational Resources We call the Gemini 1.5 Flash 55 and GPT-4o-mini [ through their public APIs. Qwen-2.5-VL [6] has publicly-availann weights, and we run the HuggingFace implementation using a server with 8x NVIDIA A100 GPUs.
Video IDs from LVBench The videos in LVBench [88 are originally from YouTube. Since some of these videos are no longer available, we include the full list of Video IDss that we were able to obtain below.
Each video has on average 15.1 questions associated with it, for a total of 1043 questions.
The full list of all Video IDs that we were able to obtain are:
-WnyRMZqV1U
16Z-XQh9jhk
2LH3JCGKEBU
2sr1HX3PbXw
2zkJFv-ro4A
3_upA09AntU
4LA_tH-VSSQ
81SbCR6p320
9-gOCOu_KGU
9tBsMSDoDgk
AeEYQ62t8hA
CgvJqGx2RFE
Cm73mal
EwskdNETNx8
FaVotIaWMEg
HFEVEGF1A8Q
JPPM28fEm10
JTa_We2MSW
JlrzSvCsIjE
KbahC-QCKU8
Mcggugol2ts
NzCO0G8AGLU
4bbpvy2xe
QB7FoIpx8os
QgwRyDv90zs
RCAqKnvu_PP
28

---

RjDrZkBwzho
S8vPx-u9p_A
SRqØweUKskM
TYDDoW
TZ0j6kr4ZJ0
VTCDQYYKA90
900.km60
Vk_AfOhtZGU
W-BnDvXXfOs
XNtNNp1AWiI
Xjf5N9S3jAA
Y1QugR7KSKg
Z4HGQL_McDQ
786xysw5Ncc
Za2Z_JRxCuk
_T2Avd3tFHc
aJI8XTa_DII
CWEnogdsw78
CXDT442T8JY
gbDR39yIs3y
hROKtPgkt08
hjoDzKesiaM
iA_69g87IIw
ihfdEFGd2dc
jp2M1hIEtsk
KBFIFQIYBVA
101A7cFNK8A
0-gLbgp2Cc8
PXD3txG2bVQ
q01CUy-gwdU
QAIRFyR6NyQ
gYMnM5b1ZIE
rk240Uu_kJQ
rp4Nkwb7dXk
skø
t-RtDI2RWQS
HH_5YbklevQ
tKIFQI9cH2c
uN9mcG@rdLY
vH1Soxg8WHo
vaL_vSdZKZo
wgB1ACG927V
xi6r3hZe5Tg
29


Reviews:
[{"review_id": "4WgfTgDjnX", "reviewer": "Reviewer_eVAj", "summary": "LLMs (and consequently VLMs) have been shown to degrade for longer inputs in “needle-in-the-haystack” scenarios of question-answering, where only a small portion of the input is relevant while the rest is considered distractions. The paper suggests a simple method to curate long input videos for VLMs by prompting the VLM to select key frames for the given input question. The authors show that the curated input improves results on short inputs (which fit inside the context window), effectively removing distractions from the input. In addition, they show that feeding this input to a small context-window model performs better than a large context-window model when fed with the entire video.", "questions": "See the strengths and weaknesses section.", "limitations": "The authors stated that the zero-shot performance of the VLM is required to be sufficiently good, but didn’t ablate on the matter. The authors should address limitations of frame-selection methods for longer relevant context, i.e. small portion of distractors (not just longer inputs).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. The proposed method shows improved results on multiple benchmarks, outperforming existing VLMs even for short inputs that fit the context window. For inputs longer than the context window, the method prunes the input, adaptively to the question, to fit in the context window.\n2. The method is simple and can essentially be applied to any VLM.\n3. Evaluation is thorough and covers 4 benchmarks, 3 VLMs, and comparisons to many methods (Table 3, top left). \n4. The authors show that the method dynamically adapts its focus depending on the type of question, achieving a similar portion of frames as human annotation (Figure 4). \n5. Ablations on different aggregation methods are also thorough, and cover both sampling based on feature similarity using SigLIP embeddings and sampling based on VLM over captions (rather than over frames).\n\nMajor weaknesses:\n1. Frame selection methods, such as the one proposed in the paper, can benefit the most in “needle-in-a-haystack” (NIAH) cases, where most of the input video is not relevant for answering the question (a large portion of “distractors”). Most long-video QA benchmarks are biased towards such cases (see [A]) and don’t reflect long-form reasoning. Frame-selection methods are therefore inherently limited, as the portion of relevant context grows, the method should adaptively select more frames, returning to the starting position where VLMs can’t truly process long inputs of relevant context. The paper does show improved results on several benchmarks with 30% and 56% relevant context frames (LVBench summarization questions and Egoschema, respectively). An evaluation on [A] with Gemini-1.5-flash should also be completed. \n2. The authors review frame-selection methods that reason over extracted captions, and state that this introduces vulnerability to the captioner's mistakes. The proposed method uses the VLM alone, eliminating this vulnerability, but assumes good zero-shot performance for the frame-selection to work. In its current form, the method is applicable to models that are already state-of-the-art. I would expect an ablation on the zero-shot performance to be enough for the method to work, followed by clear requirements\\benchmarks a new model should uphold in order to effectively deploy the suggested method.\n3. The proposed method is simple and practical, but lacks theoretical grounding and justification other than empirical results. For example, in Table 1, partitioned context aggregation is shown to outperform hierarchical context aggregation. Assuming NIAH inputs, the coarse-to-fine strategy seems like a better idea than stratified sampling (partitioned context aggregation). Indeed, as stated by the authors, this strategy is sensitive to the coarse sampling step. This can be mitigated by performing the process multiple times and finding a consensus among runs. Otherwise, an empirical study on the distribution of the resulting context frames of the method should be provided to justify the stratified sampling choice.\n4. Evaluation comparison on long videos (table 3, top right, and text) assumes that using a larger-context version of a model should result in better performance (Gemini-1.5-flash context-window to 700K tokens improves on 32K). This contradicts the premise of frame-selection, where only a small portion of frames is required, and the rest are distractors that should hurt the performance. I would appreciate the authors' opinion on the matter.\n\nMinor weaknesses:\n1. The authors should refrain from using “most relevant context” (lines 7, 9, 50, 51, Figure 2 caption) as they do not show proof of optimality.\n2. Missing ablations on different context prompts.\n3. Using the frame-selection justification of the model (j in Eq.4) to iteratively refine the prompt for the VLM can also be explored.\n\n[A] CINEPILE: A Long Video Question Answering Dataset and Benchmark, Rawal et al.", "was_revised": true, "final_justification": "The authors performed thorough experiments both in the paper and rebuttal, and addressed most of my concerns. They also related in the post-rebuttal discussion to limitations due to the size of the input memory. These limitations should be explained clearly in the camera-ready version. I therefore raise my score to borderline accept.", "rebuttal": "We thank the reviewer for their detailed comments and address each concern below:\n\n---\n### **Evaluation on Cinepile**\n\nThank you for this suggestion. We compare our Partitioned Context Aggregation approach to baseline inference on Cinepile below. We follow the same protocol as the experiments in our paper, using a 32K / 120 frame context window for both approaches.\n\n\nTable A1: Comparison of DCA on Cinepile.\n|                    | Accuracy on Cinepile |\n|--------------------|:---------------------:|\n| Baseline inference |  53.9                    |\n| DCA (Ours)         | 55.8                    |\n\nOur DCA approach therefore still shows improvement over the baseline on this challenging dataset.\n\n---\n### **Is frame-selection appropriate when there is a lot of relevant context to select?**\n\nThank you. Figure 7 in Appendix A shows that our model selects 74% of the input frames for Egoschema, and 15% of the relevant frames for LVBench. This shows that our method does indeed select the relevant context in a manner that is adaptive to the dataset. Our results on Egoschema (Figure 7 and Table 3) therefore show that when most of the input video is indeed relevant, our method is able to retain it.\n\nFinally, note that Egoschema was annotated with a \"temporal certificate” (the minimum duration of the video that an annotator must look at to answer the question) of 100 seconds (or 56% of the video), and our results in Figure 7 agree with this. Similarly, Figure 6 shows that our frame selection corresponds to the \"time reference” frames that are annotated for LVBench, with the most frames being selected for “summarisation” questions. Both these results emphasise the adaptivity of our approach in selecting both short and long relative-contexts from a video.\n\nWe will refer to Figure 7 (in Appendix A) more clearly in the main text to make this point more clear.\n\n---\n### **Zero-shot performance of VLM for frame-selection**\n\nThank you for this suggestion. Note that frame-selection for the purpose of question-answering is difficult to evaluate, as it depends on the answering model itself. From our evaluation datasets, only LVBench contains human-annotated “time-reference” frames which we use below to evaluate our frame selection.\n\nIn the table below, we extend our analysis from Figure 10 of Appendix B. We evaluate the precision and recall of our selected frames, with respect to the annotated “time-reference” frames, where a “true positive” denotes a selected frame which falls within the annotated time-references for that video, and a “false positive” does not.\n\nWe performed this analysis by varying the number of segments, $l$, in our Partitioned Context Aggregation approach. By increasing $l$, our recall increases, whilst precision remains fairly constant. We observe that the “time-reference” frames in LVBench are overcomplete, in that it is not necessary to select all of them to answer the question, which is why we can get accuracy improvements over the baseline with a fairly low recall. In fact, for $l \\leq 12$, our recall is lower than the uniform frame sampling baseline, but the accuracy is higher due to the larger precision, showing that a subset of the time-reference frames is sufficient to answer the question. We find, however, that the accuracy correlates well with an additional metric which we define, “At Least One”, which denotes if at least one selected frame falls within the annotated “time-reference” frames.\n\n\nTable A2: Analysis of frame selection on the LVBench, using the human-annotated “time-reference” frames. “At Least One” denotes if at least one of the selected frames falls within the “time-reference” frames.\n|                       | Precision | Recall | At Least One | Average Num. Selected Frames | Accuracy |\n|-----------------------|:---------:|:------:|:------------:|:--------:|:--------:|\n| Baseline, uniform sampling              |   10.0        |  12.3      |    20.3          |    120  | 50.3      |\n| Partitioned, $l = 2$  |  20.9         |   2.7     |   41.6           |   13.4  | 50.6       |\n| Partitioned, $l = 4$  |  21.5         |  4.7      |  52.4            |   20.4  | 52.0       |\n| Partitioned, $l = 8$  |  20.0         |   8.4     |   63.4           |   34.8  | 56.9       |\n| Partitioned, $l = 12$ |  19.4         |  11.5      |   69.3           | 43.6  | 60.2        |\n| Partitioned, $l = 16$ |  18.2         |  14.3      |    71.1          |  48.5  | 61.5        |\n\n\nNote that the baseline, of uniformly selecting 120 frames from the video, has the lowest precision as expected since most of the frames in the video are irrelevant to the question. Moreover, as the videos in LVBench are an average of 68 minutes long, performing uniform sampling means that we sample at least one frame within the annotated time-references only 20.3% of the time. Using our proposed method increases this to 71.1% which explains our accuracy improvement. Finally, the fact that the “At Least One” metric is less than the overall question-answering accuracy in the first two rows, also shows that the Gemini VLM is also able to correctly answer a question without using any of the annotated time-reference frames in the dataset. This may be due to errors in the annotations of time-reference frames in LVBench, or the fact that the VLM can exploit language biases to answer the question.\n\nWe will add this analysis to the appendix, and refer to it clearly from the main text.\n\n \n---\n### **Why not choose Hierarchical Context Aggregation?**\n\nThank you for this suggestion. The reviewer is correct that a coarse-to-fine context aggregation approach is effective. However, we found that in terms of accuracy/computation trade-offs, it is not as effective as Partitioned Context Aggregation. As we are not able to upload figures, we include the raw data for Figure 4, Accuracy vs Computation, below, with a line added for Hierarchical Context Aggregation.\n\nIn the revision, we will update Figure 4 to include this additional line, and to clarify this point.\n\n\nTable A3: Accuracy vs Computation Trade-off for different context aggregation methods. The table corresponds to Figure 4 in the main paper.\n|  Baseline inference  |  |  |  |  |  |  |\n|------------------------|:-:|:-:|:-:|:-:|:-:|:-:|\n| Total number of frames |  120 | 256 | 512 | 1024 | 2048 | 2700 |\n| Total number of tokens (K) | 30  | 66   | 132 | 264 | 528 | 696 |\n| Accuracy               | 50.3  | 53.6  | 56.3 | 58.1 | 58.9 | 58.9 |\n\n|  Hierarchical Context Aggregation  |  |  |  |  |\n|------------------------|:-:|:-:|:-:|:-:|\n| Total number of frames |  325 | 1104 | 1273 | 1816 |\n| Total number of tokens (K) | 83 | 284 | 328 | 468 |\n| Accuracy               | 53.1 | 58.6 | 60.3| 61.5 |\n\n|  Partitioned Context Aggregation  |  |  |  |  |  |  |\n|------------------------|:-:|:-:|:-:|:-:|:-:|:-:|\n| Total number of frames |  197 | 332 | 602 | 866 | 1515 | 2605 |\n| Total number of tokens (K) | 50  | 85   | 155 | 223 | 390 | 672 |\n| Accuracy               | 50.2  | 52.7  | 56.5 | 60.1 | 61.1 | 61.7 |\n\n|  Self-Consistency CoT  |  |  |  |  |  |  |\n|------------------------|:-:|:-:|:-:|:-:|:-:|:-:|\n| Total number of frames |  360 | 720 | 1080 | 1440 | 1800 | 2160 |\n| Total number of tokens (K) | 93  | 186   | 278 | 371 | 464 | 557 |\n| Accuracy               | 49.3  | 51.1  | 51.7 | 51.4 | 51.5 | 51.3 |\n\nNote that for our context aggregation methods, the total number of frames is based on the average number of frames selected by our method on the LVBench dataset.\n\n---\n### **Table 3 implies that longer-context is always better**\n\nThank you. We would like to clarify that as shown in Figure 4, the accuracy of baseline inference saturates around 1024 frames / 264K tokens. We will therefore update the corresponding row for the baseline of Table 3 to 264 tokens, and add a clarifying note that performance saturates after this point.\n\n---\n### **The authors should refrain from using “most relevant context” as they do not show proof of optimality.**\n\nThank you. We have revised the paper accordingly.\n\n\n---\n### **Using the frame-selection justification of the model (j in Eq.4) to iteratively refine the prompt for the VLM can also be explored.**\n\nThank you for this suggestion. We initially performed experiments with this, but did not achieve improvements beyond our current approach, which is why we did not pursue it further.\n\n---\n### **Ablations of prompts**\n\nWe ablated different prompting strategies in Tables 4 and 5 of Appendix A, and also detailed all prompts and other implementation details in Appendix D."}, {"review_id": "NQ4IY0OzFX", "reviewer": "Reviewer_buo9", "summary": "The paper proposes a \"Chain of Thought\" like approach for sampling long for video for video understanding. \nThe authors propose to use VLM for context aware frame selection as opposed to uniform sampling in long for videos.\nThe paper shows strong reasons fordoing so in terms of performance results and multiple ablation studies and analysis of them.\nWhile particular, The paper introduces a segmented approach to video understanding where the frames are first divided into non-overlapping segments and frames are chosen through a VLM. These selected frames are then used to answer the question.\n\nWhile the work shows strong reasons to use the method when it comes to performance. Their analysis of performance vs computation falls short. While the authors do present some findings, it is difficult to understand as to how their method would affect efficiency in the first place as the number of frame processed by the LLM remains the same. Second, they seem to very the number of segments as a way to increase computation but it remains unclear as to how this would be the case.", "questions": "Please clarify/fix the weaknesses as mentioned above.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Paper shows a simple and elegant approach to a difficult problem.\n2. Despite being simple, the paper is backed by strong improvements while being applicable in various areas.\n3. Strong ablation study of their own methodology covering different types of segmentation.\n\nWeaknesses:\n1. While the performance is strong, the idea of key frame extraction has been very well established. The authors do no compare their work with widely used key frame extraction methodology and only focus on mainly LLM counterparts.\n2. The efficiency argument seems unclear and even misguided as it is difficult to understand how the number of tokens being processed change at all as the authors process $m.l$ frames where $m=N/l$. Thus, I would like some clarification about the same.\n3. The claim of constant time compute of videos regardless of the number of frames as mentions in the Discussion section in section 3 seems largely false due to the same reasons as mentioned in weakness 2.", "was_revised": true, "final_justification": "The low score was due to the concerns as portrayed on the review.\n\nGiven the rebuttal, I can conclude that I had misunderstood the sole parts of the working which have been clarified during the rebuttal.\n\nWith this, I believe the impact of the work greater than what was previously given.", "rebuttal": "We thank the reviewer for their detailed comments and address each concern below:\n\n---\n### **Difficult to understand how the total number of tokens being processed change**\n\nWe would like to clarify some confusion about the total number of tokens / frames that are processed.\n\nIn Partitioned Context Aggregation (L174 to 198), the reviewer is correct that a video of $N$ frames is partitioned into $l$ segments, each of length $m = N / l$ frames.\n\nHowever, from each segment, $\\\\mathbf{x}\\_{i}$, we uniformly sample $s$ frames from it (L182 to 185), denoted as $\\\\mathbf{x}\\_{i [s]}$ in Equation 5 (*we believe this is the key detail that the reviewer missed*).\n\nTherefore, we process a total of $s \\cdot l$ frames. Note that both $s$ and $l$ are independent of the number of the video length, $N$.\nThis allows us to control the total amount of computation used to process a video in Figure 4, where we keep $s = 64$ frames constant, and vary $l$ from 2 to 42.\nFinally, note that for shorter videos where $s \\cdot l > N$, or in other words, the video is too short to partition into $l$ segments, we set $l = \\lfloor N / s \\rfloor $ instead.\n\nWe will revise the text to clarify this, and are happy to engage with the reviewer during the discussion period if this is not clear.\n\n\n---\n### **How is compute constant regardless of the number of frames?**\n\nAs described above, we process a total of $s \\cdot l$ frames, where both $s$ and $l$ are hyperparameters, and do not depend on the video length, $N$.\n\n\n---\n###  **Authors compare to LLM-based frame extraction methods, but not traditional ones.**\n\nThank you for this suggestion. As we discussed in Lines 106-111, traditional frame selection models are trained to process only a fixed, small number of input frames (ie 32 for SeViLA [68] and ViLA [59], and 64 for [A]). These models are therefore not suitable for hour-long video datasets such as LVBench which contain an average of 4080 frames.\n\nMoreover, these frame-selection models are trained for specific downstream answerer models, and need to backpropagate gradients through this downstream answerer during training. It is therefore not feasible to train such approaches for large, state-of-the-art answerer models like Gemini 1.5, GPT-4o and Qwen-2.5 VL which we use in our work.\n\nWe will clarify the limitations of traditional frame-selection techniques even further in the revision.\n\n[A] S Buch et al. Flexible Frame Selection for Efficient Video Reasoning. CVPR 2025."}, {"review_id": "OCCokzzmBp", "reviewer": "Reviewer_HQTU", "summary": "This paper presents \"Dynamic Context Aggregation\" an inference strategy for long video question-answering (QA). The proposed method tackle a problem of current VLMs, that despite the current ability of these models to process long contexts as input, they still struggle with distractors in the input sequence, affecting the performance in some tasks. The proposed approach tackle long VQA by first selecting relevant frames from the video (frame id + text justification) and then using this information to answer a given question. The method achieves state-of-the art results on 4 video question-answering datasets.", "questions": "Please refer to weaknesses for my questions and doubts. I'm currently giving a rejection to this paper. Now I do not see a strong enough contribution or really new insights. After the rebuttal and the response from the authors I will revise my decision.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Paper Strengths:\n1. The paper proposes Dynamic Context Aggregation, a combination of existing techniques that shows to be effective for selecting relevant information from videos, which shows to be useful for long video QA.\n3. Experiments show that Dynamic Context Aggregation outperforms other methods in 4 different datasets, achieving SOTA results.\n4. Most sections are well written and easy to follow\n\nPaper Weaknesses:\n1. The paper states that sometimes relevant frames are too succinct, which makes it for difficult for the second stage of the method to give a correct answer. Thus, it was found that is beneficial to include coarse uniform context (uniformly sampled frames) from the original input in addition to the selected frames. However, I’ve not been able to find this experiment to see the performance gain that is coming from adding this extra context and see if the effectiveness is coming from the good frame selection of the proposed method or from this additional context.\n2. I think that the paper has a bit of contradictory ideas, first the paper claims the well know fact that too much information affect VLMs, therefore a method to select the most important information is proposed, however the proposed method select important frames and then again add more frames from the entire video to give more context to the VLM. Maybe a better description can help clarifying this and avoid this appearance of contradictory ideas.\n3. The comparison with related works states that other works are not conditioned on the input question, and that the proposed method is simpler and elegant due to the use of a single VLM and do not rely on any external models or tools, however this approach is not entirely new, personally I’ve seen this idea of using a single VLM to get useful information by conditioning the model on the input question and provide an answer for Video QA, for example:\n[1] https://aclanthology.org/2024.findings-acl.555.pdf", "was_revised": true, "final_justification": "As I expressed in my response to the authors and in the \"Questions\" section of my original review, my main point about the paper is related to novelty, I don't see strong enough contributions from the paper to be accepted at NeurIPS. Therefore, I'm maintaining my original score of borderline reject.", "rebuttal": "We thank the reviewer for their detailed comments and address each concern below:\n\n---\n### **Ablation of uniform context**\n\nThank you for this suggestion. We already included an ablation of uniform context in Table 6b of Appendix A. In the revision, we will refer more clearly to this experiment.\n\nFor convenience, we have copied Table 6b below. We ablated the effect of the number of uniform context frames, $u$, on both Egoschema and LVBench. In both cases, the context-limit is $k = 120$ frames, meaning that the remaining $m = k - u$ frames are selected by the model.\n\n| m | u | Egoschema | LVBench |\n|---|---|-----------|---------|\n| 120  | 0  |  **75.2**         |  57.8       |\n| 88  | 32  |  74.6         |  58.5       |\n| 64  | 56  | 73.2          | **59.3** |\n| 32  | 88  | 74.0          | 56.2 |\n| 0  | 120  | 72.6          | 50.3 |\n\nOn LVBench, we achieve moderate improvements (+1.5 points) by introducing uniform context frames (first row vs third row). If we perform no frame selection, and only use uniformly sampled frames, the accuracy is substantially worse (decreases by 9 points) emphasising that good frame selection is key to good performance.\n\nEgoschema, in contrast, does not benefit from uniform context. This may be explained by Figure 7 of Appendix A, which shows that our approach is already selecting an average of 74% of the input frames in the dataset (and we therefore do not need any additional frames). The high proportion of selected frames is in agreement with the dataset’s “temporal certificate” – Egoschema was annotated [40] such that the annotator must look at, at least 100 seconds or 56% of the video to answer the question.\n\nFinally, note that the Accuracy vs Computation analysis in Figure 4 accounts for the uniform context in our method, and it clearly shows the benefit of our method over baseline inference (which is effectively only uniformly sampling frames from the input video).\n\n---\n### **Motivation for uniform context**\n\nIntuitively, uniform context is beneficial for long videos (such as the hour-long videos in LVBench), since during context aggregation, we extract relevant context from each segment of the video independently. Adding a few coarsely sampled, uniform context frames, provides a more holistic view which improves accuracy. This is not required for shorter videos, such as those in Egoschema, which may already fit within the context-limit of the model.\n\nWe will clarify this in the revision.\n\n---\n###  **Other methods which use only a single VLM for context selection**\n\nThank you. We will include a discussion of [A] in the revision, and also revise the Related Work section, specifically the “Long-context video understanding” paragraph in light of [A]. Although [A] uses a single VLM, they first compute captions as an intermediate representation of the video, and select frames based on these captions. We considered this design choice too, and ablated this in Table 2, Rows 4 and 5. Our method, based on selecting relevant frames, achieves higher accuracy than selecting frames based on their captions. Moreover, it is not as sensitive to the captioning prompt that was used, as we observed that Egoschema and LVBench require different lengths of captions to achieve high results (detailed captioning prompts are in Appendix D, Lines 1093-1094). For convenience, we have reproduced the relevant parts of Table 2 below here.\n\n|                                    | Egoschema | LVBench |\n|------------------------------------|:---------:|:-------:|\n| Select from \"concise\" captions     |  74.0          |  58.3       |\n| Select from \"long\" captions        |    72.8      |   60.4      |\n| Select directly from frames (ours) |   **75.2**        |   **61.7**      |\n\n\nThese results show that the general principle of curating the input to the VLM is beneficial, and that our instantiation of Partitioned Context Aggregation is the most effective.\n\n[A] D Romero et al. Question-Instructed Visual Descriptions for Zero-Shot Video Question Answering. ACL 2024"}]

Author rebuttal:

Meta-review: This paper presents a method to do keyframe selection to support long video QA.  The final reviews are mixed, with one accept, one borderline accept and one borderline reject.  The main source of contention seems to be the experiments (or lack thereof) to strongly support the idea as it is quite simple.

Having read the reviews, responses and reviewer discussion, the AC recommends the paper to be accepted.  The benefits outweigh the weaknesses and the work will make for an interesting addition to the program.  That having been said, the authors are kindly requested to incorporate their additional experiments and explanations throughout the discussion into the camera-ready.
Decision: Accept (poster)