MIRA: Multi-view Information Retrieval with Adaptive Routing for Test-time Long-video Comprehension
Authors: Zecheng Hao
Wayne Ma
Yufeng Cui
Shuang Li
Xinlong Wang
Tiejun Huang
Venue: Accepted by TMLR ()

Abstract: Foundational Multi-modal Large Language Models (MLLMs) have achieved rapid progress in handling complex tasks across diverse modalities. However, they still struggle to deliver satisfactory performance on Long-video Comprehension (LVC) tasks involving thousands of frames. Existing optimization strategies can be broadly categorized into LVC-specific fine-tuning, built-in token compression and training-free keyframe extraction, with the latter being most suitable for flexible deployment across various MLLMs. Unfortunately, current training-free approaches predominantly focus on query-frame relevance retrieval, overlooking other levels of visual information and the inherent heterogeneity of LVC tasks. In this work, we propose the $\textbf{M}$ulti-view $\textbf{I}$nformation $\textbf{R}$etrieval with $\textbf{A}$daptive Routing ($\textbf{MIRA}$) framework, which evaluates video frames using distinct metrics for relevance and causality, combines these scores to select a balanced pool of keyframes, and employs an adaptive feedback loop to tailor the retrieval process to different user queries, enabling more precise and sample-grained video comprehension. Extensive experiments demonstrate the advanced performance of our scheme across multiple challenging LVC benchmarks. For instance, integrating $\textbf{MIRA}$ with Qwen-2.5-VL yields performance gains of 3.5% to 13.1% on LVB, VideoMME and MLVU.

Shuang Li Published in Transactions on Machine Learning Research (02/2026)
MIRA: Multi-view Information Retrieval with Adaptive Routing for Test-time Long-video Comprehension
Zecheng Hao
heozecheng@pku.ccu.cc
School of Computer Science, Peking Universiby State Key Laboratory for Multimedia Information Processing. Peking University
Wenxuan Mat
wenzzaanbit@gmaillccm
Beijing Academy of Artificial Intelligence (BAAI)
Yufeng Cui
yfcuitbbaiicom ac..or
Beijing Academy of Artificial Intelligence (BAAT)
shuanglioi@uuaa.edu
School of Artificial Intelligence, Beihang University
Xinlong Wang
zinlong, wwng968gmaailccoom
Beijing Academy of Artificial Intelligence (BAAI)
Tiejun Huang
tjhuunn Kipku. edu.cn
School of Computer Science, Peking University State Key Laboratory for Multimedia Information Processing, Peking University
Beijing Academy of Artificial Intelligence (BAAI)
Reviewed on OpenReview: https: // openreview. net/ forum? L2b2kx08tu
Abstract
Foundational Multi-modal Large Language Models (MLLMss) have achieved rapid progress in handling complex tasks across diverse modalities. However, they still struggle to deliver satisfactory performance on Long-video Comprehension (LVC) tasks involving thousands of frames. Existing optimization strategies can be broadly categorized into LVC-specific fine- tuning, built-in token compression and training-free keyframe extraction, with the latter being most suitable for flexible deployment, across various MLLMs. Unfortunately, current training-free approaches predominantly focus query-frame relevance retrieval, overlooking other levels of visual information and the inherent heterogeneity of LVC tasks. In this work, we propose the Multi-view Information Retrieval with Adaptive Routing (MIRA) framework, which evaluates video frames using distinct metrics for relevance and causality, combines these scores to select a balancod pool of keyframes, and employs an adaptive feedback loop to tailor the retrieval process to different user queries, enabling more precise and cample-grained video comprehension. Extensive experiments demonstrate the advanced performance of our scheme across multiple challenging LVC benchmarks. For instance, integrating MIRA with Qwen-2.5-VL yields performance gains of 3.5% to 13.1% on LVB, VideoMME and MLVU.
1 Introduction
Long-video Comprehension (LVC), as one of the most challenging multi-modal reasoning tasks, has emerged critical benchmark for evaluating the advanced capabilities of Multi-modal Large Language Model
Equal contributions for this work.

---

Published in Transactions on Machine Learning Research (02/2026)
training-based
training-free
token compression
adaptive decision
LLM
LLM
LLM
LLM
Adaaptive Feelback
Relevance Retrieval
MLP
MLP
MLP
Vision Encoder
Vision
MLP
Encoder
Video Pools
Vision Encoder
Vision
Relevance Retrieval
Relesunce Ratrieval
Soneuury
Causality
Encoder
Ratrienai
Retrieval
Video Frames
User Query
Video Frames
User Query
Frames
User Query
Video Frames
User Query
(a)
(d)
Figure 1: Comparison of different optimization schemes for LVC. (a): feeding uniformly-samppie video frames into (fin-tuned) MLLMs; (b): built-in visual loken compression for MLLMs; (c): performing relevance retrieval for the video and feeding the obtained keyframe set into MLLMs; (d): MIRA framework mixes keyframe sets obtained from multiple perspectives and feeds them into MLLMs, with the specific blending ratio (video pool) regulated by the model's adaptive feedback.
(MLLM). While current foundational MLLMs (Zhang et al., 2024b; Bai el al., 2025; Chen et al., 2024b) demonstrate strong generalization across diverse multi-modal tasks, they still struggle when directly applied complex or fine-grained reasoning tasks grounded in long-form vidcos. To address this limitation, as illustrated in Fig. (a)-(c),, prior works have explored three primary routes: first, researchers have atlempted to train specialized MLLMs or fine-tune components of existing models specifically for long-video processing (Chen et al., 2024a; Shen et al., 2024; Zohar et al., 2025; Islam et al., 2025). However, due to the rich content diversity and enormous visual token budget inherent in long videos, such approaches demand prohibitively high training overhead. Moreover, the scarcity of supervised fine-tuning data often fails to adequately cover the full spectrum video semantics, limiting the generalizability of these improved models.
Furthermore, some efforts have also focused on integrating token compression techniques within MLLMs expand their effective context capacity (Cheng et al., 2025; Wang et al., 2025a; Gao et al., 2025), thereby enabling the model to ingest more visual frames. Yet, since different MLLMs employ heterogencous architoc- tures, these compression methods may lack cross-model compatibility, const raining their broad applicability. In addition, and most recently, a promising line of research has proposed externalizing keyframe extrac- tion as a training-free pre-proccssing pipeline by decoupling it from the MLLM's core architecture (Tang et al., 2025b; Ye et al., 2025). This approach leverages the strengths of pre-trained models while remain- ing architecture-ggnssicc (Wang al., 2025b; Xu al., 2025), thereby gaining increasing attention in the community.
However, current keyframe extraction methods predominantly use CLIP-like models (Radford et al., 2021) for relevance scoring, which tend to concentrate high scores on narrow temporal segments. Consequently, the resulting keyframes often fail to provide sufficient coverage of the event's 's full scope and context. More importantly, we argue that true understanding requires more than just relevance; requires capturing the causal context the frames that explain why a moment significant. These crucial pieces of evidence are largely ignored by current methods. Furthermore, this onc-sizo-fits-al approach fails to adapt to the user's query. A query about fine-grained detail needs different set of key keyframes than one about complex event, distinction overlooked in prior works that should dictate whether the extraction is localized distributed.
In this work, we attempt to address the above limitations through the following steps, as shown in Fig. 1 (d)) (i) beyond direct relevance-based based retrieval, we further construct distinct keyframe sets from the perspectives of scene relevance and causality: (ii) these keyframe sets, derived from heterogeneous sources, are then blended multiple ratios generate diverse vidco sampling pools in batch; (iii) leveraging the adaptive feedback capability of MLLMs, we enable fino- fino-grained exploration over these video pools. Our contributions are summarized follows:
2

---

Published in Transactions on Machine Learning Research (02/2026)
We introduce novel keyframe selection framework that, unlike prior works centered a singular scoring metric, uniquely integrates three complementary scoring dimensions: global-regional rele- vance, hierarchical scene context, and causal relationships. This enables our model to capture a holistic understanding of video cunentt-reoggninn not only what is relevant, but also its broader conlext and causal significance.
We establish the novel insight that by blending our three scoring metrics in varying ratios, we can construct distinct keyframe "pool" that are inherently specialized for different query intents (e.g., descriptive vs. causal questions). This reframes the challenge from generic frame selection to largeted, query-aware sampling.
We design novel, closod-loop adaptive routing strategy that leverages a model feedback loop. Instead of using a static, one-size-fis--al approach, this mechanism dynamically analyzes a user's query and selects the optimal keyframe pool on-the-fly, intelligently adapting its retrieval focus between relevance, context, and causality to best answer the specific question.
Our MIRA framework siguificantly outperforms a wide range of leading proprietary and open-source models, achieving state-of-the-art results on challenging video understanding benchmarks.
2 Related Works
Mainstream Foundational MLLMs. Recent advances in foundational MLLMs, including model families such as GPTT4o (Hurst et al., 2024), LLaVA (Li et al., 2024; Zhang et al., 2024b), Qwen-VZ VL (Bai et al., 2025), InternVL (Chen ct al., 2024b) and NVILA (Liu et al., 2025b), have demonstrated remarkable capabil- ities across diverse multi-modal tasks. These models are typically trained through multi-stage Pre-training and Instruction Fine-tuning on heterogeneous data modalities, including image, text, video, chart, docu- ment, mathematical expressions and Graphical User Interfaces (GUls), enabling them to perform complex operations such as visual question answering, cross-modal grounding, temporal localization and structured reasoning. Despite their broad generalization power, when applied to LVC, these models still exhibit a sub- stantial performance gap relative to human-level evaluation when simply applying uniform frame sampling and directly feeding the resulting frame sequence.
Training-based Models for LVC. Supervised Fine-tuning (SFT) of MLLMs remains a dominant strategy for enhancing the reasoning capability and performance on LVC tasks (Chen et al., 2024a; Shen et al., 2024). Frame-Voyager (Yu et al., 2024) leverages the prediction loss of a pre-trained MLLM to collect high-quality training data from diverse frame combinations, enabling the learning of an automated scoring component for keyframe selection. Hu et al. (2025b) annotate video samples with dual pseudo labels (spatial and temporal) to train a lightweight frame selector tailored for LVC. Zohar et al. (2025) systematically distill empirical guidelines for model training and inference the domain of IVC, based on which they propose the family of Apollo models that can effectively address long-range temporal understanding. In addition, BIMBA (Islam et al., 2025) employs state-space model with selective scan mochanisms to dynamically transmit only those most informative tokens to the language decoder. Similarly, ViLaMP (Cheng et al., 2025) integrates differential keyframe selection with weighted feature fusion, significantly suppressing temporal redundancy while retaining critical visual semantics. It is worth noting that recent works have further explored the integration of Reinforcement Learning (RL) with keyframe extraction to enable preference- aware optimization. Li et al. (2025) generate contrastive response pairs based on query-frame relevance, then jointly apply SFT and Direct Preference Optimization (DPO) to align the MLLM's outputs with human-like reasoning patterns. Inspired by the idea of Group Relative Policy Optimization (GRPO) algorithm, TSPO (Tang et al., 2025a) constructs minimal-paramete Temporal Agent trained selectively two challenging benchmarks (comprehensive temporal understanding and Boadle-in-a-Hasttade demonstrating improved long-video reasoning through lightweight policy adaptation.
Training-free Keyframe Extraction. Training- free approaches can be broadly categorized into three paradigms basod on their optimization logic: Pre-processing, Built-in Compression and Iterative Refinement. Pre-processing methods aim deliver a curated set of keyframes to the MILM prior to inference: VideoTree (Wang al., 2025b) organizes long videos into hierarchical tree structures via clustering, enabling top-down
3

---

Published in Transactions on Machine Learning Research (02/2026)
keyframe search with spatial-temporal coherence. CoS (Hu et al., 2025a) first encodes video stream into binary coding to perform pseudo temporal grounding, then feeds them into the MLLM for co-reasoning. BOLT (Liu et al., 2025a) and AKS (Tang et al., 2025b) respectively identify the cumulative distribution and local peaks along the CLIP-based relevance score curve to efficiently extract salient frames without exhaustive scanning, while Ye al. (2025) perform multi-round temporal search using keyword-driven object detection, dynamically updating the relevance distribution of the frame sequence. Nar-KFC (Fang et al., 2025) constructs interleaved image-text streams that preserve both semantic relevance and temporal continuity, enhancing the MLLM's ability to track evolving events.
In contrast, Built-in Compression methods focus on increasing the token capacity and efficiency of MLLMs during inference. SF-LLaVA (Xu et al., 2024) introduces parallel token transmission channels with varying sampling rates and pooling intensities, achieving a free lunch on performance improvement. AdaRETAKE (Wang et al., 2025a) performs adaptive token compression over both the temporal dimension and transformer layers, dynamically allocating compression ratios to enable efficient processing of thousands of frames in a single pass. APVR (Gao al., 2025) unifies query-aware semanlic expansion with adaptive visual token selection, performing hierarchical key information extraction at both frame and token levels.
Iterative Refinement methods exploit the MLLM's intrinsic reasoning and self-reflection capabilitics to dy- namically adjust the keyframe set during inference. VideoAgent (Wang et al., 2024), as an early pioneer, iteratively selects frames based on the LLM's confidence level over current answers and associated captions. Ma et al. (2025) reformulates long-video understanding as a long-document retrieval task, employing multi- stage agent interaction to progressively refine the quality of retrieved contents. E-VRAG (Xu al., 2025) explores multi-round self-reflection mechanisms within MLLMs, combined with hierarchical filtering of video content, to achieve effective long-video comprehension with relatively low computational overhead.
3 Methods
In this section, we will provide a detailed explanation of our MIRA' framework. We begin with introducing three different yet related frame scoring metrics ($3.1-33.3) ) that form the foundation for our multi-view frame retrieval. Next, combine these metrics to select keyframes that consider both relevance and causality ($3.4). Finally, present the concept of model feedback loop, designed to adapt the system to various user queries ($3.5).
3.1 Global-Regional Query-Frame Relevance Scoring
Following previous works (Tang el al., 2025b; Liu el al., 2025a), we first utilize CLIP to perform frame- by-frame feature extraction for the given video VAT RTXCXHXW (T, C, H. W are the number of frames and channels, height and width, respectively). FLT CLIPI(V e RTXD denotes the visual features processed by the CLIP vision tower with dimension D.
For a given user query, we compute the query-frame relevance score based on these visual features. While global-level semantics capture the overall scene, regional-level semantics are crucial for matching specific keywords in the user query. We capture these patch-level features as Fi. CLIPI E RTxp2x (p2 is the number of visual patches). Therefore, to achieve more discriminative scoring, we propose a hybrid scoring that integrates both global and regional relevance.
2²
F.QQ | ||
FEs Q
SR Q)
+
a)
Σ
(1)
Q
Here Q E IIIDD is the textual feature processed by the CLIP text tower and a E [0, 1] controls the relative imporlance between global and local relevance.
Prooeec URLs: https://githun http://hhhaa..ccmccc////////////
4

---

Published in Transactions on Machine Learning Research (02/2026)
Glabal mate hing
Sub-scene Window
Long Video w/ Uniform Sampling
CLIP
Query
Sti
Sij
Anchor Sub-scene
SLFX
SETZ
Ragional matckinn
J.O
Graine- Scene
Cap
Cape
Chsp
Capart
Capp
Su Q).
Top, Franc
The Belena sax saree
Counterfactual Reasoning
Video Scene Segmentation
Sn. (Sm. Q}
Gives the video seene description corrsponding to the following moments: [T most likely descraption corespocding to the scene I Please retur E.
Su
Fine-grained Sub-seene
Integrating Query-frame Relevance
8
Ref. Scones
Inf. Secnes
Py (Top, U (5,(V) near Topy (a Q1
Sc(V.)
(Metric Query-frame Relevance Scoring
Calculate Text Sim. by Tevll
Foundational MLLM
(Metric Query-scene Relevance Scoring (Metric III) Causality Scoring
Adjust Extraction Ratio
Search Video Pools
Multi-view Metrics
Model Feedbuck Loop
Concatenate Hierarchical Information
Question & Choices
Keyframes
Sample-grained Adaptive Routing
Video Pools
Figure 2: An overview of MIRA framework: for the given video and user query, sequentially perform hierarchical scene construction (left), joint relevance calculation (mid-top), representative frame and sub- scene window modeling (right-top), causality evaluation (right-bottom) and closed-loop vidco pool selection (mid-bottom).
3.2 Hierarchical Scene Segmentation For Query-Scene Relevance Scoring
Solely relying on query-frame relevance may ignore the broader event context, leading to a fragmented understanding of the video's content. Therefore, we develop the second metric based on scene segments.
Coarso-grained Scene Segmentation. Scene boundaries are identified by detecting significant shifts in the features of adjacent frames. For given scene starting index we define judgment function Jı(),, which is used to confirm whether the index interval [?, t] will be split into independent video F.-F, scene: Ji(T,)) [max;e(FAA) jE(t,** [I, < @i). Notably, to improve robustness against transient shot changes within a coherent scene, our judgment function incorporates a look-behind window. Specifically, when evaluating the current frame V,, we look back to posterior frame V... A new scene boundary is suppressed if any frame within the interval (t, maintains semantic coherence with the established scene. This prevents the model from prematurely creating
scene due to bricf cutaway, even when the feature similarity at frame V, itself is low. O represents cosine similarity threshold. The resulting scenes are denoted as Sj.
Finc-grained Sub-scone Segmentation. While coarso-grained segmentation identifics major semantic shifts, the resulting scenes are often too broad. They can simultaneously overlook subtle yet important visual changes while also containing significant. temporal redundancy. To address both issues, we perform sec-
ond, finer-graind judgment within each scene: (n( E arg, min F, jES; F
A F.F.e < On
Specifically, segment the current scene S for & times based on the expected average sub-scene length, with the splitting positions being the indices corresponding to the bottom-k similarities between adjacent frames, here Jn(i, 1) represents whether sub-scene segmentation will be performed between the 1-th and F,FFrs (t + 1)-th frames in S.. In addition, for positions with minimal visual changes (i.e. [ 2 θrr) we will skip segmentation. Eventually, we obtain sub-scenes Sn ready for query-scene relevance scoring.
5

---

Published in Transactions on Machine Learning Research (02/2026)
Query-Scene Relevance Scoring. The relevance score for any sub-scene can be directly computed by summing the corresponding query-frame scores: Sn(Sh, Q) = Eies; Sa (V,, Q) Since the frames within the same sub-scene have high similarity, we consider extracting one representative frame for each sub- scene, which not only effectively reduces temporal redundancy, but. also facilitates efficient. processing in subsequent steps.
Rep Rep = arg max (R,V,, Q).
(2)
v.esi
Comparod to query-frame relevance in the previous section, query-scene relevance scoring balances the high relevance of specific, peak moments with the broader semantic context, provided by the entire scene.
3.3 Group-based based Causal Modeling and Counterfactual Influence Scoring
The relevance scoring methods detailed in the previous sections are effective at identifying what frames and scenes visually related to a user's query. However, relevance alone is often insufficient for a comprehensive understanding, it fails to distinguish between the peak moment of an event and the crucial preceding moments that explain why or how it occurred. For example, a high relevance score might identify a building collapsing, but true comprehension requires identifying the initial explosion that caused it. To address this gap, this section introduces a third, complementary scoring metric derived from Group-based Causal Modeling. The resulting Counterfactual Influence Score designed specifically to identify frames that provide this essential explanatory and causal context, moving beyond simple relevance to capture the video's underlying narrative structure.
Constructing Sub-scene Windows. Based on the query-scene scoring, extract top relevant sub-scenes arga: max (Sn. Q) and utilize them as the anchor point to expand forward and backward in time, thereby Sn obtaining sub-scene windows composed of the corresponding representative frames. Now consider one single window constructed in this way. We denote the window by w (1((12++1) Rep' V Rep..... Vhep Rep J and the anchor scene by (21 V Rep Here w,, w \ {w,} represents the contextual set composed of neighboring sub-scenes within the window. Our goal is to identify which sub-scenes in w,, have stronger causal relationships with Wc, which can better facilitate MLLM's understanding of visual content in keyframes.
Causality Evaluation. We assess the causal influence of sub-scenes within context window in an in- teractive scenario, which consists of an inference model Te, a reference model Tref and an evaluation model Teval- Considering that text more conducive than visual signal to completing inference tasks, we respec- tively utilize a LLM,, a Captioner model and CLIPT as ne, Tref; Feval; thus projecting the visual information into the textual space. Guided by the principle of counterfactual intervention, we quantify each sub-scene's causal influence on its neighbors by masking them in turn. Concretely, we first use nref to generate captions for representative frames in each sub-scene window and request that the captions should highlight content related to the user query. For each (2k + 1--window
we can construct a set of examples with batch-size equal to 2k: {V: w, [V.), VV, E w.). Here we select a frame V; to be evaluated each time and pass the caption set corresponding to w, \ (V.} into Ty to infer the visual description of w.. Then, we compare the inferred caption with the actual caption generated by Fref before. there is a significant difference, it indicates a strong causal dependence between V, and we and we need refer to V; to better understand w., To this end, the Counterfactual Influence Score for each selected frame in the window is caleulated follows:
Sc(Vi) =
Σ
Teval (To, WroT,
Trof, W. V.)
w) V.ew,
F. Frolw [ mnae ((V))) F Fxee(w))
Σ
1 1
(3)
F F Recd(un (V.J)) s
w) V.EW
For given frame V,, we integrate the importance levels of V, in all its relevant windows [w]V, v E w.)) F (endi ((V.)) and F m.)().) denote the textual features processed by Reval for me's Ie's and Tret 's captions, respectively.
6

---

Published in Transactions on Machine Learning Research (02/2026)
3.4 The Construction of Video Keyframe Sampling Pools
We construct the final keyframe set by integrating frames from the above scoring metrics. Concretely, Query- Frame Relevance Score (SR(V) (V)) targets locally continuous moments of high correlation with the query, Query-Seene Relevance Score (SR(Sn)) identifics the most crucial summary content, while Counterfactual Influence Score (Sc) highlights frames that provide essential explanatory context. A strategic combination of these three types of frames yields comprehensive and robust keyframe selection for the video. Here we discuss two combining schemes.
Scheme 1: Global Causality. A straightforward selection strategy is to directly combine frames of high relevance and strong causality, as formulated in Eq.(4). It is worth noting that all the causal frames are drawn from context windows of highly correlated sub-scenes, hence the keyframe set implicitly encodes their representative content, even without explicitly selecting frames from these sub-scenes.
Puj = arg; max SR(Ve. Q) U arg max Sc...). V.elw.)
(4)
Scheme 2: Intra-group Causality. Another approach is to incorporate both the representative frame and the causal frame every high-valued sub-scene. Therefore, out of cach top-ranking sub-scene window that has high query-scene relevance score we select two frames: the representative frame at the central-anchor position and the causal frame that has the highest. Counterfactual Influence Score within the context window. (5) represents the entire construction process.
{ We arg max Sc(V.)lw E argy max Sn 4) V,Ewn
Pio arg; max Sn Q) U Ve
(5)
Evaluating Performance Diversity and Upper Bound in the Video Sampling Pool. It is interesting to analyze how different strategies in the pool affect the model response. Figure 3 visualizes the response behavior of seven different sampling strategies the video pool across LVC benchmarks. The horizontal axis indicates the proportion of sampling strategies that correctly answer each sample, while the grid entries denote the number of samples correctly answered by specific strategy. As we move rightward along the axis, query difficulty increases, and the intersection of correct responses correspondingly shrinks. For instance, Column 7/7 corresponds to samples that all seven sampling strategies lead to the correct answer. In contrast, Column 1/7 contains samples that only a single particular sampling strategy yields the correct answer. By comparing the numbers in column 1/7, we can conclude that the existence of those extreme sampling strategies in pool, such as pool-1 and pool-7 hold the value increasing the diversity of responses and enhancing the potential of the pool to handle difficult queries. In the ideal condition, if one could perfectly match each sample to the strategy that yields correct answer, the resulting performance would represent the upper-bound achievable by combining all strategics in the pool, which has denoted as Upper Acc in Fig.3.
The gap between the theoretical upper-bound and the actual best performance for any single sampling strategy demonstrates that even with strategic frame selection method, the long video sampling bias issue persists: ML.MSs only process a sparse slice of frames eventually, each different sampling highlights a different "story", causing the model to generate distinct answers the same query. The existence of such sampling bias reveals the shortcoming of the conventional open-loop frame selection pipeline for long video understanding: room for the adjustment of the selected frames. In light of this, further propose closed-loop solution by intograting MLLM's feedback to the inference pipeline.
3.5 Adaptive Routing Strategy based on Model Feedback Loop
We introduce two types of feedback. The first allows the model to analyze the user query and report the desired sampling before receiving video frames, while the second allows the model to say "I don't 't know" after looking at the bias-sampled video frames and requires more information. Tn other words, MLLMs guide the frame selection process in our closed-loop pipeline as an agent. We explain the details in the following.
Closed-loop pipeline 1: User Query Analysis. This pipcline considers the inherent heterogeneity of user queries in the domain of LVC, ask the MLLM to propose the desired video pool prior to the sampling
7

---

Published in Transactions on Machine Learning Research (02/2026)
740
1b22
70
1203
Text
1412
122
1683
996-1
707
4C
24
12
12
1
1332
70
8
25
20
22
24
84
47
2
12
200
1032
6
32
00033
1413
92
5
213
18/
35
38
14
1
1532
200
KG
50
24
12
1412
92
52
23
16
787
24
22
1522
106
42
1413
43
pex.7
25
1532
44
p0637
1413
57
607
53
47
377
20
10
30
07
57
47
307
20 37
52
67
57
67
207
10
(a) LVB, Qwen2.5-VL-7B
(b) VideoMME, Qwen2.3-VL-7B
(c) MLVU, Qwen2.5.
Figure 3: Distributions of correctly answered samples in multiple video sampling pools. Here the j-th column of pool-i is the number of samples answered correctly by j video pools (including pool-i) simultaneously; Upper Acc. denotes the accuracy upper-bound achiovable across all vidco pools, wherein a sample is deemed correct provided that any individual pool yields a correct answer.
process, and provides the corresponding frames adaptively. The diversity of our video pool allows the query- based routing to end up with frames suitable to answer the given query. Before providing the MLLM with sampled video frames,
let the model first parse and classify the user queries. The model is allowed to determine if looking at only a single scene is sufficient to answer the query, or multiple scenes are necessary to understand the context. Different parsing results of the user query lead to different candidates in our video pool.
Closod-loop pipcline 2: Option of Reporting Missing Information. This pipeline allows the MLLM to judge whether it received sufficient information from the sampled video frames to answer a particular query. The model is provided with additional option to refuse answering the query whenever it finds the sampled video frames are not enough for determine the answer. Using multiple-choice questions as example, we expand the option list to include the choice of "Insufficient visual information". Whenever the model chooses this option as an answer, we will increase the capacity of the video pool and repeat. the question. This pipeline guarantees that the model's final answer to the question is a fully-informed respouse.
4 Experiments
To validato the effectiveness of our framework, we choose LLaVA-Vidoo (Zhang et al., 2024b) and Qwen2.5- VL (Bai et al., 2025) as our baseline models, then evaluate on three widely adopted benchmarks in the domain of video understanding: LVB (Wu et al., 2024), VideoMME. (Fu et al., 2025) and MLVU (Zhou et al., 2024).
Long Becc (LVB) contains videos ranging from 8 seconds to 60 minutes across diverse topics, accompanied by highly detailed user queries (average length: 43.53 words). We choose its publicly available validation set, comprising 1,337 QA pairs grounded in 752 videos.
VideoMME includes 2,700 QA pairs with an average video duration of 1,017.9 seconds. Its queries are notably concise (average length: 35.7 tokens), posing a significant LVC challenge without leveraging external subtitles.
MLVU features videos up to 2 hours in length and covers multiple complex task types, including Holistic, Single-Detail and Multi-Detail LVC. We select its multiple-choice subset, consisting of 2,174 QA pairs.
Following prior works, we employ LMMs-Eval (Zhang et al., 2024a) as the evaluation toolkit. For LVB and VideoMME, we report overall accuracy; for MLVU, we report average accuracy across its multi-task setup. Unless otherwise specified, our framework extracts a fixed total of 64 keyframes per video. As defined in Eqs.(4)-(5)) we generate video pools with varying blending ratios by setting i E {8n|n E [0,8] A E N}. Specifically, for each sample, we first select the top-i frames from the perspective of relevance. Then we
8

---

Published in Transactions on Machine Learning Research (02/2026)
Table 1: Performance comparison among foundational (proprietary open-source) MLLMs, training-basee based & free methods and our framework on three challenging LVC benchmarks. Here the numbers in parentheses indicate the performance improvement compared the corresponding baseline model after introducing our framework, FPS denotes frames per second.
Method
Type
Model
Frames
LVB
VideoMME
MLVU
GPT-4o mini
Foundational
56.5
64.8
GPT-4V
Foundational
384
60.7
59.9
49.2
GPT-4o (Hurst al., 2024)
Foundational
384
66.7
71.9
64.6
LLaVA-OV (Li et al., 2024)
Foundational
LLaVA-OV-7B
32
56.5
58.2
64.7
NVILA (Liu et al., 2025b)
Foundational
NVILA-SB
256
57.7
64.2
70.1
LLaVA- Vi-dooo (Zhang al., 2024b)
Foundational
ViV----oo--B
64
58.2
63.3
70.8
Qwen2.5-Vi. (Bai et al., 2025)
Foundational
Swen2.5-VI-77
64
60.0
63.5
63.2
LongVILA (Chen et al., 2024a)
Training- based
LongVILA-7B
256
57.1
60.1
LungVU (Shen et al., 2024)
Training- baased
LongVU-7B
1FPS
60.6
65.4
Apollo (Zohar et al., 2025)
Training-base basod
Apollo-7B
2FPS
58.5
61.3
70.9
BIMBA (Islam al., 2025)
Training-
BIMBA-7B
128
59.5
64.7
71.4
TPO (Li et al., 2025)
Training-
LLAVA-Video-77
64
00.1
65.6
71.1
BOLT (Liu et al., 2025a)
Training- free
MaVA-Ov-7B
32
59.6
59.9
66.8
CoS (Hu et al., 2025a)
Training- free
LLaVA-Video-TD
64
58.9
64.4
71.4
LlaVA-Video-7B
62.7
65.3
AKS (Tang al., 2025h)
Training-frer
Qwen2.5-VL-7Bt
61.3
64.8
64.5
66.2
73.8
-LAVA-Video-78
(++6.3)
(4-2.9)
(4-3.0)
MIRA
Training- free
Qwen2.5-VL-7B
66.9
67.0
76.3
(+-6.))
(+-3.5)
(+13.11
denotes our reproduced exporimental results.
sequentially augment the selection with frames chosen according summarization or causality-oriented criteria, following the designated algorithm mentioned above. If the total frame count remains below 64 after this process, which indicates limited sub-scene diversity in the video, we supplement the selection by greedily choosing additional frames from the unselected frame set, ranked by descending relevance score, until the 64-frame capacity is reached. Specific experimental details and algorithm procedure of MIRA are provided in SAA.
4.1
Main Results
We systematically categorize existing LVC works into three groups: (1) Foundational models, further divided into proprietary and open-source variants; (ii) Training-based models; (iii) Training-free methods. For MLLMs lacking keyframe selection component, we apply uniform sampling for the given video by default. Table presents a comprehensive performance comparison among these categories and our framework.
First, compared the respective baseline models, our method achieves consistent and significant gains: +2.9% to +6.3% on LLaVA-Video- and +3.5% +13.1% on Qwen-2.5-VL-7BB across all benchmarks. Second, plug-and-play scheme, MIRA remains competitive instance, under the condition of utilizing LLa ALLVA-Ve ViVVo-7B-7B the baseline model, outperform TPO (Li et al., 2025) by 4.4% on LVB and 2.7% on MLVU. Third, when compared with similar training-free methods, MIRA also remonsttrates clear superiority: on the competitive Swen2.5-VLL-BB model, it surpasses AKS (Tang et al., 2025b) by 5.6%, 2.2% on LVB and VideoMME, respectively. In addition, it is worth noting that our approach even exceeds the performance of all proprictary models on LVB and MIVU, including GPT-4o (Hurst et al., 2024), which highlights the potential of the keyframe retrieval paradigm proposed in this work.
against training baased models. For

---

Published in Transactions on Machine Learning Research (02/2026)
Table 2: Comparison of open-loop and closed-loop pipelines on three benchmarks (64 frames))
Qwen2.5-VI-7B
LLaVA LLAVA-
Sampling Strategy
LVB
VideoMME
MLVU
LVB
VideoMME
MLVU
Uniform Sampling
60.0
63.5
63.2
58.2
63.3
70.8
Fixed Sampling
66.2
66.7
75.5
63.9
65.9
73.5
User Query Analysis
66.9
66.9
75.9
63.9
66.2
73.8
User Query Analysis
66.7
67.0
76.3
64.5
66.0
73.8
Missing Information
66.3
66.8
75.4
63.7
65.4
73.7
denotes utilizing LLM to assist in analyzing the given query.
Table 3: Ablation studies for different causal frame selection schemes described in $3.4.
Qwen2.5-VI- VL-7B
LLaVA-Video-7h
Selection Scheme
LVB
VideoMME
MLVU
LVB
VideoMME
MIVU
Global Causality
65.1
65.8
73.8
63.4
65.1
72.8
Intra-group Causality
66.2
66.5
75.5
62.2
65.5
73.0
4.2 Comparison of Open-loop and Closed-loop Pipelines
As shown in Tab.2, we compare the performance of different. sampling strategies. Here Fixed Sampling denotes using an optimal and fixed blending ratio when selecting keyframes for all user queries, which is a conventional open-loop pipeline for inference. In comparison, User Query Analysis and Missing Information refer to the two closed-loop pipelines (hat leverage feedback from the model to adjust the sampling strategy ($3.4). In Query Analysis, the sampling strategy alternates between the two best candidates in the video pool depending on whether the model determines to focus more on the highest-relevant scenes the causal content of related events. In Missing Information, models are first given 64 sampled video frames using the same blending ratio as Fixed Sampling. Tf the model chooses the additional option of insufficient information, we increase the frame mumber to 128 on those test samples (around 15%) for more visual cues, and we run the inference on them for the second time. No additional option will be given the second round.
Experimental results suggest that Query Analysis strategies can further improve upon this baseline by up to 0.7%-0.8% demonstrating the superiority of introducing dynamic adjustable feedback loop rather than relying on fixed, one-size-fits-al pool. However, consistent improvement is observed using the Missing Information strategy. Through an in-depth analysis, we discover two reasons. First, this feedback is unreli- able, as current MLLMs may not be able to effectively determine whether the visual information is sufficient. Second, adding an alterative option decreases the performance in general. Nevertheless, the performance improvement in partial cases indicates that powerful MLLMSs have the potential of understanding long videos as an agent.
4.3 Ablation Studies for Different Types of Keyframes
Causal frame selection scheme. Table makes a comparison of the two causal frame extraction schemes mentioned in Eqs.(4)-(5), here == 40 24. Compared to simply extracting a subset of frames with the strongest causality, comprehensive extraction of summary and causal frames based on sub-scene windows generally achieves superior performance, indicating that causal frames may be more suitable for playing the role of auxiliary understanding in LVC tasks.
The effectiveness of each component. In Tab.4, we investigale how keyframe composition, which is derived from different retrieval criteria, affects overall LVC performance within a single video pool. The configuration a. denotes a greedy selection based solcly on relevance scores, without sub-scene window modeling or causality-aware evaluation, i.e. FII. In contrast, configuration b. and c. introduce a moderate number of frames selected from the perspectives of summarization causality, respectively. d.
10

---

Published in Transactions on Machine Learning Research (02/2026)
Table 4: Ablation studies for mixing keyframes from multiple perspectives. Here Rel., Sum. Causal. denotes specific video pool composed of 64 frames extracted from the perspectives of relevance, summarization and causality, respectively.
Video Pool Configuration
Swen2.5-VL-7B
Rel. Sum.
Causal.
LVB
VideoMME
MIVU
LVB
VideoMME
MLVU
Uniform Sampling
60.0
63.5
63.2
58.2
63.3
70.8
a.
65.3
64.8
74.4
63.9
64.4
73.2
b.
V
V
65.7
65.4
74.4
62.8
65.5
73.6
C.
V
/
65.4
66.3
74.9
62.7
64.9
72.9
d.
66.2
66.5
75.5
62.2
65.5
73.0
represents extracting keyframes according to the second scheme in Eq.(5). Here set the total frame number of to 24. Experimental results demonstrate that selecting keyframes based on relevance alone does not yield optimal performance in most cases. Instead, the effective combination of all three keyframe types is essential to fully unleash the inherent reasoning capability ML.LMs. Results with the feedback loop can found Tab.2 and are thus not included here.
5 Conclusions
In this work, we present a test-time video comprehension framework that jointly extracts keyframes from three complementary perspectives: relevance, summarization, and causality. We demonstrate that blending these multi-view
at varying ratios allows the model to better adapt to queries with diverse intents. To this end, propose multiple closed-loop strategies that dynamically assign optimal blending ratio or keyframe capacity to each sample, aiming to harness the full reasoning potential of MLLMs. We believe that future research in IVC should focus on developing more precise control mechanisms for modulating multi-level information and enabling sample-specific allocation.
Broader Impact Statement
MIRA framework has the properties of multi-stage, training-free and closed-loop, which can support LVC tasks in both online and offline scenarios. Calling the relevant model processing interfaces locally will further enhance the security of IVC.
References
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint orXic:2502.13923 2025.
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint ar Xic:2240 10188, 2024a.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXie preprinl arXiv:2412.0522 2024b.
Chuanqi Cheng, Jian Guan, Wei Wu, and Rui Yan. Scaling video-language models to 10k frames via hierarchical differential distillation. In International Conference Machine Learning, 2025.
Bo Fang, Wenhao Wu, Qianggiang Wu, Yuxin Song, and Antoni B Chan. Threading keyframe with narra- tives: Milms as strong long video comprehenders. arXiv preprint orXiv:2505. 241668 2025.
Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark
11

---

Published in Transactions on Machine Learning Research (02/2026)
of multi-modal Ilms in video analysis. In IEEE Conference Computer Vision and Pottern Recognition. 2025.
Hong Gao, Yiming Bao, Xuezhen Tu, Bin Zhong, and Minling Zhang. Apvr: Hour-level long video under-
standing with adaptive pivot visual information retrieval. arXiv preprint orXio:2506.44555 2025.
Jian Hu, Zixu Cheng, Chenyang Si, Wei Li, and Shaogang Gong. Cos: Chain-of-shot prompting for long video understanding. arxi preprint or Kiv:2502.06420 2025a.
Kai Hu, Feng Gao, Xiaohan Nie, Peng Zhou, Son Tran, Tal Neiman, Lingyun Wang, Mubarak Shah, Raffay Hamid, Bing Yin, et al. M-llm based vidco frame selection for efficient vidco understanding. In IEEE Conference on Computer Vision and Patlern Recognition, 2025b.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perclman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, el al. Gpt-4o system card. or Xiv preprint 2xxi:2440.21277 2024.
Md Mohaiminul Islam, Tushar Nagarajan, Huiyu Wang, Gedas Bertasius, and Lorenzo Torresani. Bimba: Selective-scan compression for long-range video question answering. In IEEE Conference on Computer Vision and Pattern Recognition, 2025.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint 208.0000 2024.
Rui Li, Xiaohan Wang, Yuhui Zhang, Zeyu Wang, and Serena Beung-Levy Temporal preference optimization for long-form video understanding. ar Xiv preprint or arxi::2001. 13919, 2025.
Shuming Liu, Chen Zhao, Tianqi Xu, and Bernard Ghanem. Bolt: Boost large vision-language model without training for long-form video understanding. In TEEE Conference on Computer Vision and Pattern Recognition, 2025a.
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. In IEEE Conference on Computer Vision and Pattern Recognition, 2025b.
Ziyu Ma, Chenhui Gou, Hengcan Shi, Bin Sun, Shutao Li, Hamid Rezatolighi, and Jianfei Cai. Drvideo: Document retrieval based long video understanding. In IEEE Conference on Computer Vision and Pattern Recognition, 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkinn, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. 2021.
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understanding. arXiu preprint arXiv:2410 000:24100100 (0.117/0 17434, 2024.
Canhui Tang, Zifan Han, Hongbo Sun, Sanping Zhou, Xuchong Zhang, Xin Wei, Ye Yuan, Jinglin Xu, and Hao Sun. Tspo: Temporal sampling policy optimization for long-form vidco language understanding. arXiv preprint ar Xiv: 04369, 2025a.
Xi Tang, Jihao Qiu, Lingxi Xie, Yunjic Tian, Jianbin Jiao, and Qixiang Ye. Adaptive keyframe sampling for long video understanding. In IEEE Conference on Compuler Vision and Pallern Recognilion, 2025b.
Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nic. Adaretake: Adaptive redundancy reduction to perceive longer for video-language understanding. arXie preprint urXie:2503. 1 12559, 2025a.
Xiaohan Wang, Yuhui Zhang, Our Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understand- ing with large language model as agent. In Europeon Conference on Computer Vision, 2024.
12

---

Published in Transactions on Machine Learning Research (02/2026)
Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jachong Yoon, Feng Cheng, Gedas Bertastus, and Mohit Bansal. Videotree: Adaptive tree-based video representation for 1lm reasoning long videos. In IEEE Conference on Computer Vision and Pattern Recognition, 2025b.
Haoning Wu, Dongxu Li, Bei Chen, and Jumnan Li. Longvideobench: A benchmark for long-context inter- leaved video-language understanding. In Advances in Newral Information Processing Systems, 2024.
Chunyu Xie, Bin Wang, Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang, Dawei Leng, and Yuhui Yin. Fg-clip: Fino-grained visual and textual alignment. In International Conference Machine Learning, 2025.
Mingze Xu, Mingfei Gao, Zhe Gan, Hong- You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-fres baseline for video large language models. orXiv preprinl arXiv:2407.158/ 2024.
Zeyu Xu, Junkang Zhang, Qiang Wang, and Yi Liu. E-vrag: Enhancing long video understanding with resource-efficient retrieval augmented generation. arXiv preprint arXiv:2508.0155 015466 2025.
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report. arXiv preprint arXiv:2505. 00388, 2025.
Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Nicbles, Ehsan Adeli, Li Fei-Fei, et al. Ro-thinking temporal search for long- form video understanding. In IEEE Conference on Compuler Vision and Pallern Recogrition, 2025.
Sicheng Yu, Chengkai Jin, Huanyu Wang, Zhenghao Chen, Sheng Jin, Zhongrong Zuo, Xiaolei Xu, Zhenbang Sun, Bingni Zhang, Jiawei Wu, et al. Frame-voyager: Learning to query frames for video large language models. arxiv preprint arXiv:2/10.0322) 2024.
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. arXiv preprint arXiv:2407.122 12772, 2024a.
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Vidco instruction luning with synthetic data. arXiv preprint or 0r11:22100002113 2024b.
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint Xrxi::2406. 04264, 2024.
Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An exploration of video understanding in large multimodal models. In IEEE Conference on Computer Vision and Pattern Recognition, 2025.
A Appendix
A.1 Experimental Configuration
sypem-paramcter Scttings. Prior to hicrarchical scene construction, we perform uniform frame sampling for the given video, as illustrated in 2(). For all experiments reported in this work, we adopt default sampling rate of 1 FPS. For videos shorter than 128 seconds, we dynamically adjust the sampling interval to ensure that at least 128 frames are extracted, preserving sufficient temporal coverage for downstream processing.
In constructing first-level scenes and second-level sub-scenes, we set t" = min(t + 2,))), 0 = cos(T/44, On = cos(IT and sogment sub-scenes with an average length of frames. Specifically, for vidco scene Sj containing n frames, we apply the segmentation criterion defined in Jn(-), where the partitioning parameter k [n/4].
13

---

Published in Transactions on Machine Learning Research (02/2026)
Table S1: Detailed performance summary of video pools based on different blending ratios. Here Rel-i, Sum- jonn-Cusal--j denotes a specific pool composed of and j frames, which are respectively extracted from the frame- and sub-scene- levels, according the video pool synthesis process mentioned above. Here Bold and Underline denote the optimal and suboptimal results.
Qwen2.5- VL-7B
LLaVA- V-V---oo--7B
Video Pool Configuration
LVB
Vid
MLVU
LVB
VideoMME
MLVU
Uniform Sampling
60.0
63.5
63.2
58.2
63.3
70.8
Rel-64, Can-CCusal--
65.3
64.8
74.4
63.9
64.4
73.2
Rel-56, Sum- Cam-Causal-8
65.5
65.9
75.1
62.4
64.4
73.4
Rel-48, Cam-Causal-Te 16
66.1
65.7
74.9
63.2
65.7
73.5
Rel-40, Com-Causal-22
66.2
66.5
75.5
62.2
65.5
73.0
Rel-32, Sum- Com-Causal-ae 92
65.5
66.2
75.0
63.3
65.7
72.9
Rel-24, Sum- Com--Cusal--ae
64.6
66.4
72.9
63.2
65.9
72.9
Rel-16, Sum- Hom-Causal-ae 48
64.8
66.7
72.6
63.4
65.7
72.2
Rel-8, Sum- Com-Cousal--n 56
64.5
65.9
71.0
62.5
65.6
72.0
Rel-0, Sum- Sam-Cousal-BB 64
61.8
65.6
68.6
61.8
65.2
71.2
Table $2: Category-wise performance of video pools based different blending ratios LongVideoBench.
The performance are computed based four duration group, 15s, 60s, 600s and 3600s.
Qwen2.5-
VL-7B
VA-
Video-7B
Video Pool Configuration
15s
60s
600s
3600s
15s
60s
600s
3600s
Rel-64, Sum- Can-Clusal-O
71.4
75.6
65.0
60.3
68.8
73.3
63.1
59.9
Rel-56, Sum-Causal-8
74.6
76.7
64.6
59.8
67.2
72.1
61.7
58.3
Rel-48, Sum-Causale
74.6
76.7
65.8
60.3
68.8
72.1
62.4
59.2
Rel-40, Sum-Cousal-
74.6
76.2
66.0
60.5
68.8
72.1
62.1
57.1
Rel-32, Sum-Causal-32
74.1
76.7
64.6
59.9
68.8
72.1
62.6
59.2
Rel-24, Sum-Cousal-pu
74.1
76.7
63.6
58.5
68.8
72.1
62.1
59.4
Rel-16, Sum-Cousal-je
74.1
76.7
63.1
59.4
68.8
72.1
61.9
60.1
Rel-8, Sum- Han-Cousal-5d
74.1
76.7
62.4
59.0
68.8
72.1
62.1
57.8
Rel-0, Sum Nam-Causal-Oi
74.1
76.7
60.9
53.7
68.8
72.1
61.4
56.6
Sub-scene windows are initialized with a default length of 5 frames, truncated to minimum of frames when positioned at temporal boundaries. We retain up to 64 sub-scene windows representative frames for summarization- and causality-based keyframe selection. Here we allow representative frames to overlap across windows. To avoid redundancy, we first deduplicate the aggregated representative frame set before feeding it into The to generate descriptive captions for each sub-scene. We then apply random masking over neighboring sub-scenes and pass the masked context windows to Te for causal reasoning.
Employed Models. We utlize FG-CLIP (Xic al., 2025) for relevance calculation. Specifically, input frames resized to 224 224 before being fed into the vision tower, while questions and corresponding choices are tokenized and truncated to maximum of 248 tokens for the text tower. In Eq.(1), we set = 0.5, D = 768, 24.
For caption generation and causal inference, we utilize the light Qwen2.5-Vi and Qwen3 (Yang et al., 2025) models, respectively. For Feefs we set max pixels = 128 x 28 x 28, min pixels- = 28 x 28 and constrain caption length to no more than 25 words generally. Both models are configured with max model len 8192 and sampling parameters are performed with temperature top 0.95, top_ 20. All experiments are conducted on NVIDIA A100-SXM4-40GH and A800-SXM4-80GB GPUs.
max_pixxls
14

---

Published in Transactions on Machine Learning Research (02/2026)
Table S3: Category- wise performance of video pools based on different blending ratios on VideOMME, The performance are computed based on three duration group, short, medium and long.
Siven2.5-VL--7
LLaVA- Video-
Video Pool Configuration
Short
Medium
Long
Short
Medium
Long
Rel-64, Can-Causal-0
75.9
65.9
52.7
76.3
64.4
52.3
Rel-56, Sum-Cousal-8
77.3
65.1
55.3
77.6
63.3
52.2
Rel-48, Sum-Causal-1e
77.6
65.7
53.8
78.9
64.6
53.8
Rel-40, Sum-Causal
77.6
66.9
55.0
78.1
65.1
53.2
Rel-32, Com--Cauall-33
76.4
66.7
55.6
77.6
66.4
53.2
Rel-24, Sum- Causal-4
76.7
67.0
55.4
77.7
66.2
53.9
Rel-16, Sum- Causal-48
76.6
66.8
56.7
77.7
65.6
54.0
Rel-8, Sum- Causal-50
76.6
66.3
54.8
77.6
65.0
54.2
Rel-0, Sum-Causal
76.6
65.6
54.6
77.6
63.4
54.6
8.8
85
2898
9981
Q: What is the correct order in which the following
Fast
patterns appear the video?
Pizza parlors, the United Nations emblem, dice.
B. Dice, pizza parlors, the United Nations emblem.
3544
8858
5681
6436
5658
B681
C. Pizza parlors, dice, the United Nations emblem.
The
emblem, pizza parlors, dice.
Rel-64, Sum-C Maan-Causa--D B.
Rel-40, Sum-Causal-24: C.
X
34075
34100
85375
44075
16000
84125
Q: Why does a leopard climb on trees depicted
the video?
To prey on
food.
B. To keep an eye on the grass below.
85400
36200
36226
384126
34400
C. To search for competitors.
evade enemies.
Rcl-64, Sum-Causal-0; X
Rel-40, Sum-Causal-24;
Figure S1: The response status of different video pools on Owen2.5-VL-7B.. VideoMME. Here Rel-i, Sum- Causal-j denotes a specific pool composed of i and j frames, which are respectively extracted from the frame- and sub-scene- levels, according to the video pool synthesis process mentioned above.
A.2 Detailed Results on Three Benchmarks
This section expands upon the summary tables presented in the main paper by reporting full benchmark results across all three datasets. For each dataset, provide comparisons with prior baselines well as breakdown of the performance variations across different video pools, as shown in Tabs.S1-S3.
A.3 Visualization of Different Video Sampling Pools
Figure S1 illustrates the performance of vidco sampling pools with differont mixture ratios in responding to user queries of varying inlents. Pool Rel-64, Cam-Causal--R which consists entirely of keyframes selected from a relevance-based perspective, is better suited for addressing local, detail-oriented questions, as exemplified by the case the first row. In contrast, introducing certain proportion of keyframes derived from the perspective of summarization and causal inference enables the MILM to more effectively answer queries that require understanding of event context, such as the case shown in the second
15

---

Published in Transactions on Machine Learning Research (02/2026)
A.4 Algorithm
We provide the detailed algorithm procedure of MIRA in Alg.1. To summarize, the input video is first segmented into scenes and sub-scenes to establish a structured temporal hierarchy, enabling efficient manage- ment of long sequences. Then the frame-level and sub-scene-level relevance scores are computed with respect to the user query, providing an initial ranking of potentially informative segments. Highly relevant repre- sentative frames are extracted and used as anchors to form localized temporal windows, capturing context around critical events without processing the full sequence. Next, captions are generated for representa- tive frames, while neighboring segments are masked to assess predictive consistency. This causality check selects the context. frames that contain crucial information to understand the event in the video. Finally, multiple video pools are generated, each blending segments at varying ratios. A closed-loop routing strategy iteratively selects the most coherent path, yielding the final MLLM-generated responses aligned with query objectives.
Algorithm Multi-view Integrated Retrieval with Adaptive Routing (MIRA) Input:
the given videos V (1:N),,1: and user queries Q(1:N), the inference, reference and evaluation model Te, Trefs Teval for causality modeling
Output: the MLLM response set A(I.N)
1: # Hierarchical Video Scene Construction
Apply JI(.), Jn(-) for V ((::)),::1 to sequentially establish video scenes Uh s and sub-scenes U S (
3: # Relevance Calculation
4: Calculate the joint relevance scores U SR (V (.).:: Q(1) frame by frame according to (q.(1)
5: # Sub-scene Window Modeling
6: Calculate the relevance scores of video sub-scenes U SR(S Q(a)
7: Extract reprosentative frames from U ((() based on (2.
Construct corresponding sub-scene windows N (we, wo)() by utilizing top-k relevant sub-scenes (rep-
resentative frames) as contral-anchor points # Causality Evaluation
10: Generate captions for representative frames and randomly mask neighboring sub-scenes Wn,
11: Employ to predict the visual description at the central-anchor point corresponding to cach set of mask data
12: Calculate the causal scores of relevant representative frames U Se Se(V( VV (0).T E {wal(s) ac- cording to E3.
13: # Closed-loop Routing Strategies
14: Generate M video pools P (1:M)).(:M)) with different blending ratios based on Eqs.(41-(5)
15: Based on the existing video pools, choose a specific closed-loop pipeline to obtain the final model response set A(I:N) of sample gramularity
A.5 Implementation Detail and Analysis of Closed-loop pipelines
In this final section we elaborate more on the implementation details and the analysis of the proposed closed-loop pipelines: User Query Analysis and Missing Information.
User Question Analysis. We adopt the following prompt to ask the MILM classify the user query into "single" or "multiple" category: Determine if the following question be answered by viewing single scene from the video, or if it requires understanding events and relationships across multiple scenes. Oulput *eraclly* one lowercase word: "single" or "multiple". Do not include any other leal, punctuation, or explanation. For example: Question: What color the car? + single Question: What is the woman with the pink hat wearing > single Question: Why did the person run away? -> multiple Question: What is the order of the following event? > multiple Question: < |placeholder >. The "< !placeholder) >" string will be replaced by the actual user query. No video frames are input into the model at this stage. After
16

---

Published in Transactions on Machine Learning Research (02/2026)
the classification, will determine the blending ratio of i and for high-ranking frames and representative frames. For user queries that are identified as "single", we select more high-ranking frames, whereas the number of representative frames increase when the question is classified as "multiple".
Missing Information. We add an additional option in the candidate list of the user query. For instance, the original question and candidates are: What the color of the car? A. Red B. Gray White D. Black + (64 sampled video frames) and the modified input becomes: What is the color of the car? A. Red B. Gray C. White D. Black E. Insufficient visual information + (64 sampled video frames). If the model still responds with one of the original four options, test samples as the sufficient set. If the answer is 'E', we sample more frames from the vidco and modify the input at a second time: What is the color of the car? A. Red B. Gray C. White D. Black + (128 sampled video fromes). These samples are included in the insufficient set. We decide to end at the second round for the simplicity the pipeline. We have discovered interesting phenomenon when we analyzed the imperfect results of this pipeline, as shown in Tab.2. First of all, we observed that the new option influences the MLLM's decision even when the model thinks that the visual information provided is enough. As result,, the model performs worse the sufficient set when given the first modified input. Therefore, even though increasing the sampling frames on the insufficient set improves the performance, the overall improvement is less significant. Second, by visualizing the samples identified as insufficient, we realized that the model cannot fully judge whether it receives enough visual clues from the video to answer the question. We hope that reinforcement learning on the model could help improve its capability of determining whether is given proper question. We will leave this as the future work.
record its answer, and the inference ends. We call the union of these
17


Reviews:
[{"review_id": "ra2NO5rmNL", "reviewer": "Reviewer_nvXo", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors introduce a training-free long-video keyframe selection method that improves VQA by mixing global and regional query–frame relevance, hierarchical scene-based representativeness, and a mask-based influence score that measures the impact of removing nearby context on the predicted caption for an anchor sub-scene. The core method builds multiple candidate keyframe pools with different mixture ratios, and uses a lightweight adaptive routing loop with optional handling of \"insufficient information\" settings to pick a pool in a query-adaptive manner. They report consistent gains on LVB, VideoMME, and MLVU using the LLaVA-Video-7B and Qwen2.5-VL-7B models as backbones.", "claims_and_evidence_yn": "Yes", "claims_explanation": "1. The analysis of diversity in the sampling pool (illustrated by Figure 3) is helpful in showing that different video pools and mixture ratios help across different subsets, and is good motivation to support adaptive routing instead of fixed ratios. \n2. There is a thorough comparison against fairly comprehensive baselines, with both training-based and training-free methods and frontier models like GPT-4o. \n3. Consistent and clear improvements with two backbone MLLMs models across three benchmarks, although the closed-loop vs open-loop (sampling strategies) comparison in Table 2 is less convincing. The findings do reflect the central contributions claimed, nonetheless.", "audience_yn": "Yes", "audience_explanation": "The problem of long-video understanding is timely and practically important, with MLLMs being increasingly deployed for video analysis. The core methodology is thoughtfully described and motivated at each step. This clarity is appreciated in making this work accessible and readable to a broader audience while bridging video understanding with a smart retrieve-and-generate framework, which may have insights relevant to several sub-communities (e.g. video-language models, RAG, efficient inference).", "requested_changes": "1. The “causal” framing does not seem to be empirically supported, since the formulation in Eq. 3 is predictive influence / consistency under masking measure, a proxy for mutual information in caption space rather than actually identifying causality. It does seem a bit strong to term this as a “Causal Influence Score,” to this effect. Ideally, the authors should distinguish what is proven (that this framework / the schemes in Section 3.4 have  utility for QA), from what is intended but not demonstrated (a causal explanation). \n2. A table reflecting computational overhead for MIRA would be informative for practitioners, with metrics like wall-clock timing compared with uniform sampling and single-metric retrieval, the number of frames initial sampled in the raw pool for CLIP scoring, masking calls, etc.\n3. It would be helpful to compare MIRA with strong training-free baselines on fixed total inference cost rather than just matching frame count. Again, this would inform overhead for practitioners. \n4. Please clarify whether Fixed Sampling in Table 2 uses the single best pool from Table S1 or if it is an average across pools.\n5. Describe the intuition behind the threshold choices in Appendix A.1; otherwise, these seem a bit arbitrarily chosen. If there is prior literature or some ablations / sensitivity analysis motivating this, please add or cite it. \n6. Scaling analysis showing how performance varies as total frame budget increases from 64 to 128 (and maybe to 256) would inform whether the is a change in benefit of this method relative to baselines. \n7. “Mask-based causal inference” (page 2 and Figure 2): it would be good to clarify that this refers to masking neighboring frames, not image-level masking.", "broader_impact_concerns": "N/A -- perhaps it would not hurt to include a brief statement on potential misuse risks (e.g. privacy concerns) and how performance can vary across diverse contexts. This could nicely fit with the authors' motivation for the inclusion of the \"Missing Information\" handling, as well as how this relates to the notions of \"causality\" and relevance in video understanding and generation through this method.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "X2hpGC6eFo", "reviewer": "Reviewer_32Gn", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "MIRA is a training-free keyframe selection framework for long-video question answering with existing multimodal large language models. The paper argues that standard query–frame relevance retrieval tends to over-focus on isolated moments and fails to provide sufficient contextual and explanatory frames for long-horizon video reasoning.\n\nThe method constructs three types of frame-selection signals: (1) query–frame relevance based on global and regional CLIP similarity, (2) hierarchical scene and sub-scene summarization where scenes are segmented and representative frames are selected based on aggregated relevance, and (3) a mask-based influence score that estimates how neighboring sub-scenes affect an anchor scene by measuring caption changes under masking. These signals are combined in different ratios to form multiple candidate frame pools.\n\nAt inference time, the framework uses simple MLLM feedback to select among these pools, either by prompting the model to classify the query type before sampling or by allowing a “missing information” option that triggers resampling with more frames. Experiments on LongVideoBench, VideoMME, and MLVU with LLaVA-Video and Qwen2.5-VL show consistent over uniform sampling and other training-free baselines.\n\n**Strength:**\n- Practical, training-free wrapper: the approach can be applied at test time on top of existing MLLMs without any fine-tuning.\n\n- Clear motivation for multi-view frame selection.\n\n- Reasonable modular design.\n\n**Weakness:** \n- The method relies on a long, multi-stage pipeline with **many hand-tuned, interdependent hyperparameters** spanning relevance scoring, scene segmentation, causal windowing, and pool routing. \n- How robust is the method with respect to different prompt format? (asking model feedback, etc.)\n- The paper’s use of the term “causality” refers to a heuristic sensitivity-to-masking score rather than a formal causal notion, which may be misleading given the lack of an explicit causal model or identifiability assumptions.\n- Figure 2 is not reader-friendly. The text color makes it very difficult to read. Furthermore, the middle block does not give a clear difference between sub-scene scores/frame scores and other than that. Overall this figure does not help me understand the paper.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper provides details analysis of their method and compare with other SOTA approaches.", "audience_yn": "Yes", "audience_explanation": "Test-time inference for VLLM is an important topic in this area", "requested_changes": "See Weakness. Please address it in your new version.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Uo0SrwHDIw", "reviewer": "Reviewer_LyCH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes MIRA, a multi-view information retrieval framework for long-video comprehension (LVC) with multimodal LLMs. MIRA combines three relevance scores (global + patch-level relevance score, a scene relevance score, and a caption-based causal influence score derived via counterfactual ablations) to generate multiple sampling pools that capture complementary “views” of the video, and a closed-loop model feedback mechanism to refine sampling when needed. \n\nThe authors evaluate MIRA on LVB, VideoMME, and MLVU benchmarks using two MLLM backbones, reporting consistent improvements over existing approaches. \n\nThe work aims to address the challenge that no single sampling strategy captures all aspects of long videos, and positions MIRA as a robust, multi-view alternative.\n\n\nStrengths\n-The paper clearly articulates the different approaches to LVC, and why single-strategy sampling is brittle.\n-Methodological modularity: MIRA is training-free, model-agnostic, and easy to integrate with existing MLLMs.\n-Causal-influence scoring is novel within LVC: Although related to counterfactual leave-one-out methods, the specific caption-based formulation is new in inference-time video sampling.\n-Comprehensive benchmark coverage: Evaluation spans three LVC datasets and two MLLMs (LLaVA-Video (Zhang et al., 2024b) and Qwen2.5-VL (Bai et al., 2025)).\n\nWeaknesses\n1. The causal scoring metric risks missing temporally-distant causal events\n2. Insufficient attribution of improvements to model features, with the majority of improvement from a single relevance score\n3. Limited evaluation of causal claims of the causal relevance score\n4. Missing key long-video benchmark: EgoSchema\n\nFurther information provided in the \"Requested Changes\" section", "claims_and_evidence_yn": "No", "claims_explanation": "See requested changes.", "audience_yn": "Yes", "audience_explanation": "This work will be of interest to ML practitioners who work with video comprehension systems.", "requested_changes": "1. The causal scoring metric risks missing temporally-distant causal events\nThe causal influence module evaluates frames only within local windows around sub-scenes chosen for high relevance. This creates a bias that events that occur much earlier in the video may never be evaluated by the causal module. Many long-horizon tasks require reasoning over such precursors (for example, the placement of “Chekhov’s gun” in the beginning of a film may be causally critical to its firing much later).\n\nWhile I don’t anticipate that this will be solvable with the current framework, it is a meaningful limitation which I recommend that the authors acknowledge and discuss how future versions of MIRA might incorporate long-range causal dependencies. If there are clear ways this can be readily improved, or the if accuracy of causal assessment at long durations can be assessed, I would recommend this be included in the manuscript.\n\n\n2. Insufficient attribution of improvements\nWhile the paper has used ablation studies to investigate the effectiveness of the different elements of the keyframe composition, the results do not isolate whether performance gains are attributable to the model-feedback loop. An additional ablation study should be done with the model-feedback loop removed. Furthermore, in Table 4 it seems that the vast majority of performance improvement is from relevance sampling, with only very modest gains from adding the causal and summarization pools. This should be acknowledged in the text, as one of the main claims is that *multiple* pools is important for LVC,  and I wonder if the impact of the causal/summarization pools might be improved via changes to the model feedback loop that focus on identifying additional relevant frames for these pools.\n\n\n3. Limited evaluation of causal claims\nThe paper uses the term “causal” to refer to the frames extracted from the causal evaluation metric, but does not provide any direct evaluation against ground-truth causal event structure, qualitative examples showing that the causal module retrieves genuinely explanatory frames, or tests involving long-range causal dependencies. As written, the “causal” score reflects semantic consistency under frame ablation, rather than causal reasoning in a strict sense. While this approach is clearly well-motivated by counterfactual reasoning, a more cautious framing, or better yet, additional validation of the causal claims, would strengthen the work.\n\n\n4. Missing key long-video benchmark: EgoSchema\nThe evaluation omits EgoSchema, which is a widely used for long-horizon video-language understanding. Even if the authors choose not to evaluate on it, the dataset should be discussed in related work with justification for its exclusion.\n\n\nMinor concern:\nFigure 3 (correct-ratio heatmap) is difficult to interpret. I recommend expanding or clarifying the caption and axis descriptions.", "broader_impact_concerns": "The authors may wish to add a broader impact statement regarding the uses of LVC for e.g. security purposes.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)