ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding
Authors: Yiyang Zhou
Yangfan He
Yaofeng Su
Siwei Han
Joel Jang
Gedas Bertasius
Mohit Bansal
Huaxiu Yao
Venue: NeurIPS 2025 poster ()
Keywords: Video understanding; Multi-agent framework; Reflective reasoning; VLA alignment; Video reasoning
Primary area: applications
TL;DR: ReAgent-V enables reward-driven, multi-agent video understanding with dynamic reflection and frame selection.

Abstract: Video understanding is fundamental to tasks such as action recognition, video reasoning, and robotic control. Early video understanding methods based on large vision-language models (LVLMs) typically adopt a single-pass reasoning paradigm without dynamic feedback, limiting the model’s capacity to self-correct and adapt in complex scenarios. Recent efforts have attempted to address this limitation by incorporating reward models and reinforcement learning to enhance reasoning, or by employing tool-agent frameworks. However, these approaches face several challenges, including high annotation costs, reward signals that fail to capture real-time reasoning states, and low inference efficiency. To overcome these issues, we propose ReAgent-V, a novel agentic video understanding framework that integrates efficient frame selection with real-time reward generation during inference. These reward signals not only guide iterative answer refinement through a multi-perspective reflection mechanism—adjusting predictions from conservative, neutral, and aggressive viewpoints—but also enable automatic filtering of high-quality data for supervised fine-tuning (SFT), direct preference optimization (DPO), and group relative policy optimization (GRPO). ReAgent-V is lightweight, modular, and extensible, supporting flexible tool integration tailored to diverse tasks. Extensive experiments on 12 datasets across three core applications—video understanding, video reasoning enhancement, and vision-language-action model alignment—demonstrate significant gains in generalization and reasoning, with improvements of up to 6.9%, 2.1%, and 9.8%, respectively, highlighting the effectiveness and versatility of the proposed framework.

ReAgent-V: A Revard-Driven Multi-Agent Framework for Video Understanding
Yiyang Zhou Yangfan He Yaofeng Su'
Siwei Han' Huaxiu Yao
Joel Jang?
Gedas Bertasius' Mohit Bansal UUNC-Chapel Hill, University of Washington yiyangai@cs. unc edu, huaxiuccs unc.edu
Abstract
Video understanding is fundamental tasks such as action recognition, video reasoning, and robotic control. Early video understanding methods based on large vision-language models (LVLMs) typically adopt single-pass reasoning paradigm without dynamic feedback, limiting the model's capacity to self-correct and adapt in complex scenarios. Recent efforts have attempted to address this limitation by incorporating reward models and reinforcement learning to enhance reasoning, by employing tool-agent frameworks. However, these approaches face several challenges, including high annotation costs, reward signals that fail to capture real-time reasoning states, and low inference efficiency. To overcome these is- sues, we propose ReAgent-V. V, novel agentic video understanding framework that integrates efficient frame selection with real-time reward generation during inference. These reward signals not only guide iterative answer refinement through a multi-perspectiie reflection mechanism--ajjusting predictions from conserva- tive, neutral, and aggressive viewpoints-but also enable automatic filtering of high-quality data for supervised line-tuning (SFT), direct preference optimization (DPO), and group relative policy optimization (GRPO). ReAgent-V is lightweight, modular, and extensible, supporting flexible tool integration tailored to diverse tasks. Extensive experiments 12 datasets across three applications- video understanding, video reasoning enhancement, and larioon-nnnuggg-aeee aatiign model alignment- demonstrate significant gains in generalization and reasoning, with improvements of up to 6.9%, 2. %, and 9.8%, respectively, highlighting the ef- fectiveness and versatility of the proposed framework. Our code are available https: //githu. com aalaammminnngiiaaaaeaaaaiil 1ab Laaiiinng-aRRReeettt..
Introduction
Video understanding, as one of the core tasks in computer vision, is widely applied in scenarios such as action recognition, video reasoning, and robotic control. Although significant progress has been made in recent years through the application of large vision-language models (LVLMs) and multimodal transformers, most existing methods adopt a static reasoning paradigm- the model generates predictions directly from the input prompt and video in single pass, without reflection or reward-based feedback to guide the reasoning process [TO]
is,
6 This design limits the model's adaptability in complex scenarios that demand iterative reasoning task-specific feedback. For example, in tasks involving multi-step reasoning or reward- wewerd-based evaluation, the model must reason step by step, verify intermediate results through reflection, and potentially use reward signals to assess the accuracy of each step. Without such mechanisms, static reasoning hampers the model's 's ability to self-correct and optimize its reasoning trajectory.
56
Equal contribution.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Catropy-Calibrated Frame Selection
The sequence of color changes when the person picks the paper in the video?
HH
Question
Tool
Video
Tool Factory
Selected Video Frames
Prediction Flow
Application
Final Answar
Inference
Soarc:
Initial/Refieed Answer
Assrnay
Target Agent
Critie Agent
Cribial Questions
SFT
DPO
Evaluation
GRPO
Reflection
Reppott
Conservative
Neutral
Aggressive
Training
The system first selects relevant video frames based the imput question through the entropy calboay-calibrate frame selection module and invokes various tools from the tool factory to assist in reasoning. The target agent generates an initial answer using the selected tools and input context, which is then critically evaluated by the critic agent through questioning and scoring, ultimately producing a comprehensive feedback report. The target agent can then revise its answer from three perspectives-conserraaa report and updated context. In addition to generating standard reasoning outputs, high-scoring data identified during the reflection process based on the feedback report is stored for training algorithms such as SFT, DPO, and GRPO, thereby further enhancing model performance.
Figure 1:
Overview of the ReAgent-V V framework:
neutral, and bggressive-basedd on the
To improve video reasoning performance, previous research has mainly relied on two strategies: expanding high-quality annotated datasets for supervised fine-tuning (SFT) and introducing external reward models or reward templates to enhance the base model through reinforcement learning [TO]
33
20 While both approaches offer certain benefits, they also face notable limitations. On the one hand, obtaining high-quality annotations is often prohibitively expensive, limiting the scalability of methods that depend increasing the volume of labeled data. On the other hand, using pretrained reward models for offline evaluation fails to capture the model's real-time reasoning state during inference, which can lead suboptimal strategies or even reinforce flawed reasoning patterns. Although reward templates provide some form of real-time feedback, they are typically static and lack the adaptability needed to handle complex tasks respond dynamically to changes during the reasoning process. For example, reward templates are typically predefined and are limited to providing feedback on multiple-choiee answers or responses that follow specific templates, which makes it challenging to evaluate the reasoning process underlying a model's answer. More recently, approaches based on multi-mode debate tool-agent frameworks have been proposed enhance reasoning capabilities [40] 9 [52] However, these methods are often time-consuming and inefficient, with limited functionality, and they lack the ability provide reward signals during inference.
54
To address these limitations, we propose an innovative agent-based video understanding framework, ReAgent-V, shown in Figure The idea of framework not only utilizes dedicated frame selection module for efficient video understanding during inference, but also generates real-time reward signals. These signals help the model refine its answers, and the same time, they can be used to filter high-quality data for SFT, DPO [33], and GRPO [34), enabling continuous optimization through learning during inference. To further enhance this process, ReAgent-V incorporates an intelligent reflection mechanism that prompts the target agent to revise its answers from conservative, neutral, and aggressive perspectives. where conservative only adjusts the final answer, neutral updates entities in the scene based on the input context, and aggressive modifies both the reasoning steps and the involved entities. This multi-perspective evaluation helps mitigate biases caused by model overconfidence and leads to more robust and diverse answer refinement. Additionally, ReAgent-V V is designed with simplicity and extensibility mind. provides modular architecture that allows users to customize the base models of both the target agent and the critic agent, as well as the task

---

templates. ReAgent-V V supports variety of applications, such as enhanced video understanding and video-based model rewarding. The former aims to improve the video reasoning capability of the existing base model, while the latter leverages reward feedback during the reasoning process to collect high-quality data, which then used for training to further enhance the model's performance.
In conclusion, the key contribution of this work is ReAgent-V, the unified video understanding agentic framework capable of providing real-time rewarding during inference. Through extensive experiments on 12 datasets or tasks across three major applications-vider understanding, video
LLM reasoning, and -asoon-anguuge action model alignment. ReAgent-V V achieves performance improvements up to 6.9%, 2.1 11, and 9.8%, respectively. Additionally, our study highlights the contribution of the frame selection mechanism in ReAgent-V V and the intelligent reflection system, both of which are essential for improving the reasoning efficiency of the agent framework, reducing overconfidence bias, and enhancing the accuracy of reward feedback.
2 ReAgent-V
To improve video understanding and support dynamic reward feedback throughout the inference process, we propose a lightweight, general, and extensible agent framework, ReAgent- V. illustrated in Figure [ ReAgent-V enhances the reasoning process of the target agent through three key stages: entropy-calibrated frame selection, tool-augmented reasoning, and intelligent reflection with dynamically generated reward signals. First, the framework selects task-relevant video frames using an entropy-guided sampling strategy (2.Ip. reducing redundancy and constructing a concise semantic context. Next, the target model performs initial reasoning by invoking appropriate external tools (82.2). Finally, based on the reward signals provided by the critic agent, the target agent revises its initial answer from multiple perspectives conservative, neutral, and aggressive- teteeeyyybbyyyyy improving the model's adaptability and robustness (2.3) Detailed descriptions of each component are provided in the following sections, and the overall process is outlined in Algorithm 1
2.1 Entropy-Calibran Frame Selection
In this subsection, introduce the keyframe selection strategy adopted in ReAgent-V V.
Rermullin value.
Score Entrayy Score
Common methods typically rely on generating captions for each frame or directly querying LVLMs such as GPT-4o select keyframes that are most relevant to the prompt (49] [40]. However, such approaches
cue aeation)
0.0
inefficient and slow in practice. Others focuses on CLIP-
15
30
Frame Index
100
150
200
Figure 2: Frame selection analysis (VidcoMME VideolD: 24i4ncHuf6A, QuestionID:0005-2 shows entropy, CLIP score, and ECRS across frames; red lines highlight the most relevant frames.
based selection (30 48). where frames are chosen based on the CLIP score between each frame and the prompt. While efficient, this approach often fails on questions like "Does the person wash their hands before or after placing the cup?". where many frames contain a person, resulting in high CLIP scores even for irrelevant moments. This leads to the inclusion of redundant frames that are semantically unrelated to the question. We conducted a statistical analysis this phenomenon, with detailed experiments reported in Appendix D One example is shown in Figure 2 where the frames relevant to the question are manually annotated (as indicated by the red lines) after watching the video. We can observe that selecting frames solely based on the CLIP score tends to result in redundant frames.
To address this limitation, we propose a metric called the Entropy-Calibrated Calibrated Relevance Score (ECRS), which jointly considers the semantic relevance of each frame to the query and the information entropy of each frame-i.e. the amount of information it contains as illustrated by the orange line in Figure 2 Specifically, given a video V {fi.f2.... f2...................... , fvv consisting of N frames and query Q. we first extract feature embeddings for each frame S; and the query Q using shared CLIP encoder:
e, = (f)) q q= (Q):(Q).
(I)
where e; and q represent the embedding vectors of the i-th frame and the query, respectively. The semantic similarity between each frame and the query is then computed by the cosine similarity e. g Te, |||l]
quantifies how well each frame fi aligns with the query Q. To incorporate frame-level
3

---

Algorithm 1 ReAgent-V Video Understanding Pipeline
Require: Video V; Query Q; Toolset T; Prompt for critics agent Qei Reflection prompt for target
agent (ley by: [a]
Ensure: Final answer Annal; Evaluation report E
Stage 1: Frame Selection (2.1
2: Compute ECRS using Eqn. 3
3: while there are frames meeting the threshold ECRS am do:
4: Save the current round's s selected frames Fm.
5: Recompute ECRS in set Fm.
6: Obtain the selected frame set F
7: Stage Tool-Augmented Reasoning
$2.2
8: Target agent select tools T C T
9: Obtain tool information via Ti(Q, F), where T, T'
10: Provide the target agent with F, Q. and R to obtain the initial answer Ag-
11: Stage Evaluation and Reflection ($2.3)
12: if Critic agent rejects Ao then
13: Generate questions {q: J
14: Select new tools from T, update Rupdute R. and generate evaluation report E.
15:
Target agent generate reflected answers and confidence score ((()),(()))) for & (lesbnsta) from Rupdate, Q, F and
16:
if min(plo), (p()))))))))),,,))))))))){)))) /p())= > 0.6:
17:
Atnal + merged reliable parts of ( based on E and Q, where & E ftertn.ta tn. tu} else: Afinal t- AA),, where & = arg (tott p(t)
18:
end if
20:
return Ajnal E
information entropy, we denote the entropy of the i-th frame as H,. For color images, we compute the entropy separately for each RGB channel and take the average across channels:
H
Σ
ΣeS
( P
loga Pj.
CE(R,G,B)
(4) where Pi. represents the probability of pixel value in channel of the i-th frame. Finally, the ECRS for each frame is defined as:
84 H.
ECRS,
Σ H
where is the total number of frames selected in the current frame set. This score jointly considers the similarity between each frame and Q, as well as the amount of information contained in each frame. After obtaining ECRS, frame selection is performed iteratively. We set base threshold and an iteration index starting from 1, frame fi is selected if:
ECRS; > k am .T.
The term k-a m serves as scaling factor for the selection threshold. The scaling factor in each iteration allows the threshold to increase exponentially, thereby ensuring that frames selected in subsequent iterations have higher ECRS. In each iteration m, set candidate frames Fm. {f1 f2..... fe} is selected, where denotes the number of frames chosen in the current round. For each frame in F.. we recalculate its score ECRS; using Eqn. (3) and apply Eqn. (4) again to select new frames. This process continues until additional frames meet the threshold, in which case return the frame set selected from the previous iteration. If the number of frames is below 32, we select the non-repetitive frames with the highest ECRS, from the previous round's frame to complete the selection. As shown in the lower part of Figure 2, our method consistently increases the ECRS of key regions after iteration while suppressing those of irrelevant areas, leading more accurate frame selection.
2.2 Tool-Augmented Reasoning
After selecting the key frames, to enhance the reasoning capability of ReAgent- V in video under- standing tasks, we design tool-augmented iterative reasoning mechanism. This mechanism allows

---

target agent to dynamically select and apply tools based on task requirements, iteratively refining its understanding through multiple rounds of information exchange, resulting in higher accuracy and interpretability. Specifically, given the user query Q, the selected key frame set F, and a tool set T, the target agent proactively selects subset of tools T S 'T based on its reasoning needs, and applies them to the user query Q and the selected key frame set F, for each tool T; in T', generating intermediate tool results R; = (Q, F). These intermediate results, along with Q and F', are fed into the target model, producing an initial answer Ao.
2.3 Evaluation and Multi-Perspeetivee Reflection
After the target agent generates initial answer Ao based on the selected frames F, query Q. and tool outputs R. the critic agent evaluates the quality of Ag. If the answer is deemed unsatisfactory. the critic agent first generates set of sub-questions {91 ................................ localize potential errors. Using these sub- sub-questions, the critic agent supplements the original toolset T' with additional tools as needed and applies them to F obtain updated tool outputs Rupdate- The critic agent then uses this updated information to produce process reward for the initial answer in the form an overall evaluation report E, which includes both a scalar reward score and structured feedback on initial answer Ao (see Appendix C for examples of evaluation reports).
. qr based on the context help
Based on the evaluation report, the target agent performs reflective reasoning from three strategic perspectives: conservative (fc), neutral (t,,), and aggressive (fa). The conservative strategy focuses solely on adjusting the final answer, the neutral strategy updates the relevant entities the scene (i.e., objects or elements related to the question) based on the given context, and the aggressive strategy revises both the reasoning steps and these entities. Prompt templates for each strategy are provided in Appendix B Each strategy independently generates revised answer A and an associated confidence score p(*), where t {tc, tn ta The confidence score represents the model's estimated probability of the revised answer given the context:
(C)) = P(A F, Q, Rupdate- E).
(5)
Finally, if all confidence scores p(t) exceed a predefined threshold, the target agent aggregates the three revised answers {A(tz), A(t,) A(t,), extracts their common components, and removes inconsistencies to generate the final answer Afmal- Otherwise, the system selects the answer with the highest confidence.
Experiments
In this section, demonstrate the effectiveness of ReAgent-V V across three distinct video-related applications by addressing the following key questions: (1) Can ReAgent-V V enhance performance in video understanding? (2) Does the feedback reward mechanism in ReAgent-V V effectively select data that improves the reasoning capabilities of vidco LLMs? (3) Can the preference data generated through ReAgent-V improve alignment in preference optimization? (4) Are the frame selection and reflection mechanisms in ReAgent-V V beneficial?
3.1 Applications and Experimental Setups
This section outlines three applications and describes the experimental settings, evaluation bench- marks, and baseline models. The applications include video understanding, video LLM reasoning, and Aasion-language-actionn (VLA) model alignment. Further details are provided below.
Video Understanding. This application evaluates the effectiveness of ReAgent-V in enhancing base models for video understanding. We apply ReAgent-V V to range of base models with varying sizes and architectures, including LLaVA- S-IV---Vee--- 7Bd72- [56] and the Qwen series [46] 38 (Qwen2.5-
VL-7B/72B and Qwen2-VL-7BB) and assess their performance across six widely used benchmarks: LongBench 39 NextQA 45), EgoSchema [11] LVBench [39], MLVU [59], and VideoMME (I].
To provide a comprehensive evaluation, we also include popular proprietary models B7] open- source video-based LVLMS 4 21 55
15 and general video agent frameworks [40] For
fair comparison, we re-evaluate all baselines (except proprietary models) using similar number of video frames. During evaluation, both the target and critic models are set to the ReAgent-V-enhanee V-enhanced model itself. Detailed descriptions of the task prompt templates, the tool list. benchmark datasets,
5

---

Table I: A comparison of ReAgent-V with state-of-the-art video-language models and general video agent frameworks across different benchmarks. The results baselines are copied from [59] 9 40 30 "w/ sub" in VideoMME means that the model's input during evaluation includes the video's subtitle.
Model
Frames
LongBench
NestQA
Egonchema
LVBench
MLVU
VideoMMEE,, suh
subset
Short
Medium
Long
Overall
GPT-4o [4]
384
34.7
64.5
80.0
70.3
65.3
71,9
SharGGrT/Vidk-BB
1
16
39.7
46.4
48.2
36.3
35.0
39,9
VslooCha227B 21
16
56.7
39.3
47.9
48,3
37.0
33.2
39,5
4aaAA-NeTT-Vibo
154
16
28,
49.4
43.0
36.7
43.0
InemmV1-2.1-888 5
64
31.4
64.2
Kangarno (LLaMA2-88)
56.0
Owen2-VL.7DD [3]
32
44.7
72.1
55.3
32,6
$4.6
65.7
51.2
43.9
53.6
Owen2.5.VIL.n1
32
46.2
73.5
33,0
56.3
54.1
LLaVA-Video-78 50
32
45.8
73.2
56.6
32.8
56.7
71.4
$3.6
43.5
56.1
Long-LLaVA-77
52.0
LLaVA-Vidko72B
32
60.7
75-4
74.7
38.7
72.8
78.0
63.7
59.6
67.1
1wen2.5-VLL-22 2
34
62,5
76.8
75.4
39.3
73.1
79.4
71.2
71.
74.1
RMMBA-l.aVAQQenn227)) 13
64
76.9
60.3
50.1
Vidonkgent
49.0
56.0
VideoMMeAAgen
72
51.2
70.8
62.8
58.2
55.3
64.2
52.7
57.4
LLaVA-Video-78 ReAgenr-V
34
53.1
74.6
008
35.2
58.8
72.7
$4.8
46.7
57.9
-L- ReAgent-V
33
46.4
74.9
35.6
56.1
70.3
48.6
58.3
Qwee2.5.V1 ReAgent-V
40.5
73.5
49.8
60.7
LLaVA V2d ReAgent-V
38
64.9
83.2
75.1
40.7
73.3
78.2
70.9
71.4
73.5
Owen2 VL-722 ReAgges-V
32
66.4
84.3
762
41.2
74.2
80.1
72.3
72.9
75.1
and baseline models are provided Appendix A and
Video LLM Reasoning. This application focuses on enhancing the reasoning capability of Video LLMS using ReAgent-V V. Inspired by prior work [42) that demonstrates the effectiveness of using smaller, high-quality datasets to improve LLM or VLM reasoning, we leverage ReAgent-V as data selection mechanism to identify valuable training samples. Specifically, we apply ReAgent-V to video samples from the Video-R1-260k [00 dataset, using the scores in E output by ReAgent-V V during inference as the sample importance score. We retain videos and their original questions with importance scores lower than 5 (out of 10), these cases tend to require more reflection during inference and are more challenging for the model. As such, we consider them more informative for training. The selected samples are then used to fine-tune the model using GRPO 344. Following the experimental setup in [0 99, we conduct comprehensive evaluations on 6 video reasoning datasets. Detailed descriptions of the benchmark datasets and baseline models provided in Appendix A VLA Alignment. This application investigates whether ReAgent-V V can go beyond improving general video understanding to effectively benefit VLA alignment. Following (57), we aim to enhance the alignment of VLA models by using Re ReAgent-Ve Vas the reward model, replacing the template-bate reward function originally proposed in [57] Specifically, we apply ReAgent-V to evaluate randomly collected rollouts from Open in the SIMPLER environment (23). These rollouts are assessed across multiple dimensions success failure, task completeness, stability, and accuracy accuracy-resultinnn resulting in an overall evaluation score. These scores are then used to align the model's behavior via trajectory- wise preference optimization (TPO) [57].
We evaluate the aligned models across four types of generalization: in-domain generalization, subject generalization, physical generalization, and semantic generalization, each consisting of multiple subtasks. Additionally, we benchmark performance against a diverse set of baselines trained under various paradigms and reward models, including OpenVLA-SFE Vpen and OpenVLA TPO with GRAPE. All experiments are conducted under consistent settings, with each training paradigm using the same number of trajectory pairs ensure faireess. Detailed descriptions of task setups. model configurations and hyperparameters are provided in Appendix
3.2 Main Results
We present the results ReAgent- V across three applications: video understanding (Table T). video LLM reasoning (Table and VLA model alignment (Figure(3). Overall, ReAgent-V consistently yields substantial improvements across all tasks, achieving performance gains of 6.9%. 2.1 %, and 9.8% in the respective applications. A detailed analysis for each application is provided below. a
Video Understanding. video understanding, ReAgent-V effectively enhances the performance of open-source LVLMs across various parameter sizes and architectures. For instance, without introduc-

---

Table 2: Model performance under different optimization strategies using Qwen2.5-VL--7B as the base model across various benchmarks.
Video Reasoning Benchmark
Video General Benchmark
Models
Frames
Amount of data
VSI-Bench
VideoMMMU
MMVU
MYBench
TempCompass
VideoMMM
Original
16
47.8
57.4
SFT
16 260%
31.8
47.4
61.3
59.4
69.2
52.8
Vanilla GRPO
16
116k
32.3
45.8
60.6
60.9
69.8
53.8
GIIN ReAgeniV
33.1
47,9
ing additional frame inputs, ReAgent-V V achieves average performance gains of 3.2% and 6.9% on multiple benchmarks compared to the original Qwen2.5-VL-72B and LLaVA-Video-72B Video-72B, respectively. Notably, for LLaVA-Video-72B ReAgent-V attains performance comparable GPT-4o despite using significantly fewer input frames, highlighting its effectiveness and efficiency. Furthermore, at the same parameter scale, ReAgent-V surpasses other agentic methods such VideoMemAAent [9] while using fewer frames and maintaining higher efficiency.
Video LLM Reasoning. In the video LLM reasoning, ReAgent-V outperforms vanilla GRPO by a relative average of 2. 2,, while using only 45% of its training data. This demonstrates the effec- tiveness of data selection in enhancing video LLM reasoning, as well as the strength of ReAgent-V V in guiding that selection. Moreover, ReAgent-V surpasses the SFT strategy, which was trained on the full 260k video-image dataset, achieving an average relative improvement of 4.3% across all reasoning and general benchmarks. These results collectively highlight the value of leveraging evalu- ation reports generated during ReAgent- inference as rewards for guiding effective data selection.
VLA Alignment. In the VLA align- ment task, ReAgent-V achieves a 9.8% overall improvement over the second- best baseline, GRAPE (which uses a template reward), on SIMPLER under the same setting. Specifically, on the original task, ReAgent-V outperforms GRAPE by 9.0%. These results suggest that, compared GRAPE- -which re- lies on predefined reward function to evaluate trajectory
(22
m.1
414
410
312
201
294
Qpeiii spernie Balgeer/
sola LAOCLD
2st. pellow
the lowd
Figure 3: Comparison of ReAgent-V with Open and other reward method on the same data on the Simpler-Env environment.
provides more accurate rewards for the trajectories generated by VI.A models, thereby achieving better alignment.
3.3 Quantitative Analysis
In this section, we conduct comprehensive analysis to evaluate the effectiveness of incorporating entropy-calibratee frame selection and the perspppperee reflection mechanism.
Analysis of Frame Selection Mech- present the
anism. In Table average par-sample inference time with and without frame selection, along with the corresponding perfor- mance differences across three bench- marks for various models. The re- sults demonstrate that the frame selec- tion strategy introduced in ReAgent- consistently yields substantial per- formance gains across LVLMs with diverse architectures and parameter scales, while simultaneously reducing inference time. This highlights the ef- fectiveness of our entropy-call callbbraedd frame scoring strategy selecting key frames that are both question-relevant and informative. Fur- thermore, we compare the performance of different video agent frameworks, and it is clear that
Table 3: Analysis on frame selection mechanism in ReAgent- V, "Time" denotes the average per-sample inference time on the corresponding benchmark.
Model
Selection
LongHench
VideoMME
EgoSchemn
Acc. Time Accy%) Time Acc.(%) Time
VidenAgent (GPL-4)
41,4
48.6
54.9
VideoMemAAent (GPT-4)
51.2
122.3
57.4
143.7
62.8
152.6
LLaVA-78
53.1
$2.6
21.38
38.99
57.9
56.5
28.38
33.07
60.8
57.8
34,86
41.96
QwenVL-778
46.4
23.02
60.7
27.33
61.9
44.08
LLaVA-728
64.9
64.3
83.24
68.19
75.1
74,8
65.38
89.37
73.5
73.1
77.53
$6.53
Qwen2.5-72B
75.0
87.61
69.73
68.21
75.1
7

---

LongBench
EgoSchema
Table Reflection analysis ReAgent-V.
Corr.
Corr.
Percnntage (%)
80
Percentape (%)
60
Model
Reflection|1 Lnggench VideoMME EgoSchema
53.1
$7.9
60.8
52.9
59.4
Qween- VL-78
46.4
58.3
56.4
&
RoAgent
f. ReAgont
46.1
58.2
55.8
Figure 4: Comparison of four reflection strategies
LLaVA-Vidcon BBBcoo
64.9
75.1
(tas tu ter and ReAgent-V), where Corr. Rate
x
64.2
73.1
74.8
denotes the frequency of answer revision and Corr.
Qwen2.5.VI.. 5-V1-728
66.4
75.1
76.2
Ace indicates the accuracy of those revisions.
65.9
75.0
75,9
Question
What the tolling when the burger placed in the upper right video corner the end the video first appears? steak began to be candwiched between two pleces bread.
Beef with spices came from Ruaia to Germany. burgers spread throughout the States. The sbandardiztioon hamburgers.
Smawrr pliced
Carblnen
deaee cante-dke snussuree
Structured FirectrreFeeeaacss regort
Imsepnnatioon the
OCR
DET
Hiwees: speciil t'arna satieencdd
Arowes
larget Agent
To folinesatiin sluad "Hac"" cast secuns
and White Cos pii---hppedd building-
tie
commerciaa dtandarciioaion
hamburgan euggrr the dtanes
Initial answer 1B
Ortis Questiees
leech untarver the tar
spenat
Thas witl nonthe coinect interratatin weaee terpprrtimomeee cu curdoon..
Aspreeeiee
Muli-Agent Evaluation
Scate Ressodd
Add Tool:
Model
Vauni Auir ars E-weaer....
Colie www.bauuatooono... 2.05..
Spech 2,55 reavrr Liga 4e24.55 resortt...
Reflection
Critle Agent
Tarret-Agom
koove 8006
Final answer
Figure A case study demonstrating how ReAgent-V enhances video understanding through iterative reasoning and tool use (see additional examples in Appendix C)
ReAgent- V achieves significantly higher efficiency than other general video agent frameworks, further validating its effectiveness.
Analysis of Reflection Mechanism. To analyze the reflection mechanism in ReAgent-V, con- ducted experiments across models with different architectures and parameter sizes. Specifically, we compare the performance of ReAgent-V with and without the reflection module video under- standing tasks. The version without reflection corresponds to the pipeline in Figure 4 where the agent directly performs cool-augmented reasoming based on selected frames. As shown in Table 4 incorporating the reflection module consistently improves the performance of various models on video understanding benchmarks. Furthermore, Figure 4 we analyze the individual contributions of the three different perspectives conservative, neutral, and aggressive within the reflection process. Experimental results show that under the aggressive strategy, which allows modifying both the target agent's 's reasoning process and final answer, the corrected answers have the lowest accuracy. This may be because the model, trying to follow the reflection template, mistakenly alters previously correct reasoning steps. In contrast, the conservative strategy, which only permits changing the final answer without modifying the reasoning, yields better correction performance, indicating that errors are more casily fixed when the original reasoning is preserved. By integrating three perspectives, ReAgent-V V effectively mitigates errors caused by incorrect reasoning modifications while retaining the ability to correct mistakes, thereby enhancing the stability and overall accuracy of the reflection mechanism.
3.4 Case Study
To evaluate fine-grained video understanding, we present a case from the VideoMME dataset involving the question "What is the video conveying when the burger first appears in the upper right corner?" scene marked by burger, "1920" timestamp, and castle-like White Castle structure as is shown in Figure 5 Initially, using OCR and object detection, the model answered ("The steak began to

---

be sandwiched between two pieces of bread"), missing key symbolic and temporal cues. Reflection via three sub-questions introduced a caption model, enabling semantic interpretation that revised the answer to C. recognizing the 1920 and castle as symbols of fast food's standardization. Multi-agent evaluation showed diverse views: the aggressive agent favored D (industrialization), while neutral and conservative agents supported C. Reward scores confirmed gains in reasoning specificity and visual alignment. This case demonstrates the complexity of symbol-rich, temporally grounded video QA and underscores the value of dynamic tool integration, reflection, and reward-driven evaluation.
4 Related Work
Multimodel LLMs for video understanding. Large vision-language models (LVLMs) have been widely applied in video understanding [14
37
4
[2]
55
5 25 These models integrate tools like
Optical Character Recognition (OCR) 32 Automatic Speech Recognition (ASR) [50], video object
detection [29] 17]. and video captioning 33
47 to better interpret complex video content. Recent work has applied these tools tasks like video question answering by fusing visual, textual, and auditory features [30 [6]. However, carlier approaches are ceatic-producing answers in single pass without iterative refinement-limitinn their ability to handle multi-step reasoning and generation to dynamically select relevant video segments [35). and multi-agent systems have emerged [40] (9) enabling codaalit-speecialized agents collaborate for real-time refinement and error correction, mimicking human-like iterative reasoning. Reflective reasoning further helps agents adapt outputs
4
g
self-correction 30 [6]. To overcome this, newer frameworks employ
based on new information [53
Nevertheless, most existing methods lack real-time reward feedback and struggle with refining predictions during inference. They also tend to overprocess video frames, lacking efficient frame selection 44 D. and often rely heavily on the model's internal capacity or require retraining for effective reflection. In contrast, our framework, ReAgent-V, V, introduces unified multi-agent system with real-time, inference-aware reward signals and novel multi-penn petttiiprrppeeivve reflection mechanism. This enables dynamic refinement during inference and facilitates automatic selection of high-qualit, outputs for fine-tuning. Consequently, ReAgent-V achieves greater adaptability, interpretability, and performance across video understanding tashs-without the need for costly annotations or static reward templates.
Data-Centric VLM Reasoning Strategies. To enhance reasoning accuracy, methods like supervised fine-tuning (SFT) 77 and reinforcement learning [43] with human-defined reward models such as direct preference optimization (DPO) (33] and group relative policy optimization (GRPO) (34], have been developed. These approaches rely on fine-tuning base models with large annotated datasets and reward models that prioritize human-alignes aligned outputs 7 33 34]. While promising, these strategies face critical challenges: annotating high-quality temporal data, such as action sequences or causal reasoning, is labor-intensive and costly, limiting scalability. Additionally, pre-trained reward models, typically used offline, cannot adapt to model's real-time reasoning during inference 44
28 34 18
This becomes problematic in dynamic scenarios where feedback needs to be nuanced and responsive to the evolving reasoning process [5T] 19 I. As result, these reward models may fail to provide timely corrections improve in woaldworld, dynamic tasks [60]. These limitations highlight the need for more adaptive and scalable approaches to enhance model performance in complex environments.
reinforce erroncous reasoning patterns, hindering the model's 's ability to
Conclusion
We present ReAgent-V, a unified, agentic framework for video understanding that introduces inference- time reward generation and multi-perspectivy reflective refinement. By combining lightweight entropy-calibran frame selection with dynamic feedback, ReAgent-V moves beyond static, single- pass reasoning and enables effective tool aug-enegeetee self-correction. Experiments on 12 datasets across three applications-viian action model cerumment-dmmoossiin overhead. We hope this framework inspires further exploration into efficient, self-improving agent systems capable of operating reliably in video understanding and reasoning.
understanding, video reasoning enhancement, and vasion-language- consistent performance gains (up to 9.8%) with minimal

---

Acknowledgement
Y.Z., S.H. and H.Y. are partially supported by Cisco Faculty Research Award and Amazon Research Award.
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-237366 2022.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang. Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923, 2025.
[3] Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308 14508, 2023.
[4] Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. Sharegpt4video: Improving video understanding and generation with better captions. Advances in Neural Information Processing Systems, 37:19472- 19495, 2024.
[5] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open- multimodal models with model, data, and test-time scaling. arXiv preprint a1210:2412.05271, 2024.
[6] Zhe Chen, Jianman WU, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 24185-24199 2024.
[7] Pádraig Cunningham, Matthieu Cord, and Sarah Jane Delany. Supervised learning. In Machine learning techniques for multimedia: case studies organization and retrieval, pages 21-49. Springer, 2008
[8] Songtao Ding, Shiru Qu Yuling Xi, and Shaohua Wan. A long video caption generation algorithm for big video data retrieval. Future Generation Computer Systems, 93:583-595, 2019.
[9] Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: memory-augmented multimodal agent for video understanding. In European Conference on Computer Vision, pages 75-92. Springer, 2024.
[10] Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning milms. arXiv preprint arXiv:2503.21776, 2025.
[11) Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint 10Xi 2024.
[12] Randy W Heiland, M Pauline Baker, and Danesh K Tafti. Visbench: A framework for remote data visualization and analysis. In International Conference on Computational Science, pages 718-727. Springer, 2001.
[13] Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501.13826, 2025.
10

---

[14] Aaron Hurst. Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprini arXiv:2410.212776 2024.
[15] Md Mohaiminul Islam, Tushar Nagarajan, Huiyu Wang, Gedas Bertasius, and Lorenzo Torresani. Bimba: Selective-scan compression for long-range video question answering. arXiv preprint arXiv:2503.099 095500 2025.
[16] Soyeong Jeong, Kangsan Kim, Jinheon Baek, and Sung Ju Hwang. Videorag: Retrieval- augmented generation over video corpus. arXiv preprint arXiv:2501.05877 2025.
[17] Licheng Jiao, Ruohan Zhang, Fang Liu, Shuyuan Yang, Biao Hou, Lingling Li, and Xu Tang. New generation deep learning for video object detection: A survey. IEEE Transactions on Neural Networks and Learning Systems, 3306):3195-3215, 2021.
[18] Zixuan Ke, Fangkai Jiao, Yifei Ming, Xuan-Phi Nguyen, Austin Xu, Do Xuan Long, Minzhi Li, Chengwei Qin, Peifeng Wang, Silvio Savarese, al. A survey of frontiers in Ilm reasoning: Inference scaling, learning to reason, and agentic systems. arXiv preprint arXiv:2504.0003 2025.
[19] Somnath Kumar, Yash Gadhia, Tanuja Ganu, and Akshay Nambi. Mmetagent: Multi-modal crit- ical thinking agent framework for complex visual reasoning. arXiv preprint arXiv:2405.18335 2024.
[20] Cheng Li, Jiexiong Liu, Yixuan Chen, and Yanqin Jia. Video-vot-rl: An efficient video inference model integrating image packing and aoe architecture. arXiv preprint arXiv:2503. 15807, 2025.
[21] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint 230505335. 2023.
[22] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, al. Mybench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195-22206, 2024.
[23] Xuanlin Li, Kyle Hsu, Jiayuan Gu, Oier Mees, Karl Pertsch, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real-world robot manipulation policies in simulation. In 8th Annual Conference on Robot Learning, 2024.
[24] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122. 2023.
[25] Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting cong-context video imput. arXiv preprint arXiv:2408. 155522220..555,, 2024.
[26] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang. Jie Yang. Qing Jiang, Chunyuan Li. Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision, pages 38 38-55. Springer, 2024.
[27] Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video lims really understand videos? arXiv preprint arXiv:2403.00m 00476, 2024.
[28] Zijun Liu, Peiyi Wang. Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. timerence-time scaling for generalist reward modeling. arXiv preprint trxiv:2504.02495. 2025.
[29] Shengyu Lu, Beizhan Wang, Hongji Wang, Lihao Chen, Ma Linjian, and Xiaoyan Zhang. A real- time object detection algorithm for video. Computers & Electrical Engineering, 77:398 2019.
11

---

[30] Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually-alie aligned retrieval -aggeeetde long video 2024.
comprehension. arXiv preprint arXiv:2411.
[31] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances Neural Information Processing Systems, 36:46212 12-44444, 2023.
[32] Ravina Mithe, Supriya Indalkar, and Nilam Divekar. Optical character recognition. International journal of recent technology and engineering (WRTE), 2(1):72-75, 2013.
[33] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 01553328-537441 2023.
[34] Zhihong Shao, Peiyi Wang, Qihao Zhu, Rumxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint 2tt1:22022033000, 2024.
[35] Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634 8652, 2023.
[36] Mingkang Tang, Zhanyu Wang, Zhenhua Liu, Fengyun Rao, Dian Li, and Xiu Li. ClipAcaption: Clip for video caption. In Proceedings of the 29th ACM International Conference on Multimedia, pages 4858-4862, 2021.
[37] Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05536 2024.
[38] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing language model's I's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[39] Weihan Wang. Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406. 080350 2024.
[40] Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. In European Conference on Computer Vision, pages 38-76. Springer, 2024.
[41] Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal Ilms to 1000 images efficiently via hybrid architecture. arXiv preprint arXiv:2409.02880 02889999 2024.
[42] Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung-Chine Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Sota with less: Mcts-guided sample selection for fata-efficient visual reasoning self self-improvemeen arXiv preprint arXiv:2504.077 00934444444 2025.
[43] Marco A Wiering and Martijn Van Otterlo. Reinforcement learning. Adaptation, learning, and optimization, 12(3):729, 2012.
[44] Xiaobao Wu. Sailing ai by the stars: A survey of learning from rewards in post-training and test-time scaling of large language models. arXiv preprint arXiv:2505.026: 02686, 2025.
[45] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 9777-9786, 2021.
[46] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 1:X1v:2412.15115. 2024.
12

---

[47] Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Just-, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-seale pretraining of visual language model for dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10714-10726, 2023.
[48] Zeyuan Yang, Delin Chen, Xueyang Yu, Maohao Shen, and Chuang Gan. Vea: Video curious agent for long video understanding, arXiv preprint arXhv:2412.10471, 2024.
[49] Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, et al. Re-thinking temporal search for long-form video understanding. arXiv preprint (xXi::254.002259 2025.
[50] Dong Yu and Lin Deng. Automatic speech recognition, volume 1. Springer, 2016.
[51] Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in neural information processing systems, 222233334365511 2021.
[52] Lu Zhang, Tiancheng Zhao, Heting Ying, Yibo Ma, and Kyusong Lee. Omagent: A multi- modal agent framework for complex video understanding with task aivide-and-ooquer arXiv preprint wriv:2406.166600 2024.
[53] Wenqi Zhang, Ke Tang, Hai Wu, Mengna Wang, Yongliang Shen, Guiyang Hou, Zeqi Tan, Peng Li, Yueting Zhuang, and Weiming Lu. Agent-pre Learning to evolve via policy-level reflection and optimization. arXiv preprint arXiv:2402. 17574, 2024.
[54] Xingjian Zhang. Siwei Wen, Wenjun Wu, and Lei Huang. Tinyllav- Towards smaller Imms for video reasoning. arXiv preprint urXiv:2504.09644 2025.
[55] Zhang, B Li, H Liu, Y Lee, Gui, D Fu, J Feng, Z Liu, and Li. Llava-next: A strong zero-shot video understanding model. 2024.
[56] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma. Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint 2rXiv:24/0.027133 2024.
[57] Zijian Zhang, Kaiyuan Zheng, Zhaorun Chen, Joel Jang, Yi Li, Siwei Han, Chaoqi Wang, Mingyu Ding, Dieter Fox, and Huaxiu Yao. GRAPE: Generalizing robot policy via preference alignment. In Workshop on Reasoning and Planning for Large Language Models, 2025.
[58] Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, al. Mmvu: Measuring expert-level multi-discipline video understanding. arXiv preprint arXiv:2501. 12380, 2025.
[59] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding, arXiv preprint arXhv:2406.04266 2024.
[60] Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprini 190%00990..08933. 2019.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We accurately describe the scope and contributions of this paper in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitaions of our work in appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
14

---

Justification:
The paper does not contain any theoretical results or formal proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the experimental setup, training details, and hyperparameters in Appendix A to ensure reproducibility.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [No]
Justification: All datasets are public datasets. We are working towards making our code available upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
httpollliigggddde/ooooooo weweeeeeeeooeSbumiiioooe ssionlolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed experimental setup, training details, and hyperparameters in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification:
We discuss this in experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
16

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computational resources used in the experiments are indicated in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: The research conducted in the paper fully conforms with the code ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impacts in appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release any models or datasets with high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All original papers and codebases appropriately cited in this paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
18

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
19

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLMs not used important, original, non-standard component of the core method.
Guidelines:
The answer NA means that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
20

---

A Experimental Setup
A.1 Dataset and Baselines
A.1.1 Video Understanding
The video understanding benchmarks evaluated in this work encompass six diverse and high-quality datasets, each meticulously curated to target distinct and critical aspect of video-language com- prehension. These benchmarks collectively span a wide range of tasks lockks-including long-horizon reasoning, egocentric perspective analysis, multimodal alignment, and fine- aineggrined offering comprchensive and rigorous evaluation of model capabilities across various real-world video scenarios.
Designed to evaluate long-horizon reasoning over extended video contexts, requiring models to maintain semantic coherence across many frames.
LongBench [3]
NextQA [45] Focuses temporal reasoning and causal understanding by presenting questions about action sequences and their logical consequences.
EgoSchema 31 A diagnostic benchmark for egocentric videos, assessing the ability understand
first-person perspectives and actions.
LVBench (39] Tailored for multi-modal alignment, especially in complex settings where audio, text, and vision interplay is critical.
MLVU [59] Emphasizes multi-task long video understanding, pushing models to perform a range of tasks such as captioning. question answering, and temporal ordering.
VidcoMME [I]: A comprehensive benchmark that integrates various subtitles and OCR-and is especially suited for evaluating fine-grained visual-semantic alignment and symbolic reasoning in videos.
To rigorously assess the performance of ReAgent-V. V, we compare it against broad array of baseline models that span different capabilities and design philosophies. Proprietary closed-source models include GPT-4o [1), known for its advanced multimodal reasoning, and 1.5ini-1.5-Pro [37], optimized for long-context multimodal input. Among open-source video-language models, the evaluation includes:
ShaveGPT4Video 4]: Enhances caption-driven video question answering by integrating ShareGPT. style conversational supervision into multimodal video training. leverages large-scale user- generated captions and dialogues to align vision and language effectively, enabling better general- ization to open-endee open-ended video queries.
Vhae [21]: chat-centric video-language model built to support multi-turn, conversational interactions grounded in video content. It emphasizes interactive understanding, with capabilities for dialogue continuity, temporal grounding, and multi-modal alignment, making it ideal for assistive agents and education scenarios.
LLa Video [54]: An extension of the LLaVA framework adapted for vidco inputs, available in both 7B and 72B parameter versions. It employs frame-wise and temporal fusion techniques to convert video sequences into language-alignnn visual question answering and summarization tasks.
embeddings, demonstrating strong performance on
Qwen2 [2] and QWen2.5-VL [2] families: Developed by Alibaba, these models exhibit high performance on both image and video understanding benchmarks through advanced multi-modal aligmment. Qwen2.5-V LL particularly excels in handling long-context and dense risual-textual reasoning tasks, supported by a unified languae architecture.
Intern VL-2.5 [6] A high-performing open-source model that scales both the architecture and training data to improve generalization. incorporates vision- -anguala alignment techniques, dense region grounding, and optimized pretraining routines support both short and long video tasks with high efficiency.
BIMBA-LLAY 15]: Introduces selective scan compression mechanism tailored for long videos, where it prunes redundant frames while preserving key semantic content. This compression- training improves inference speed and memory usage while maintaining accuracy on temporal and contextual reasoning tasks.
21

---

Kangaroo [25]: Designed specifically for long- context video input, Kangaroo adopts hierarchical attention and memory- efficient tokenization strategies to process hundreds of frames. It is particu- larly effective for document-level video understanding, such as meeting summarization sports analysis.
variant of LLLaVA optimized for efficient multi- processing. It inte- grates temporal coherence constraints and cross-frame attention mechanisms, making it capable of capturing nuanced motion patterns and temporal dependencies for improved video QA and description.
Long-I.LaVA
VideoAgent [40]AA agent framework for long-form video understanding that mimics human cognition through LLM-guided reasoning, CLIP-based retrieval, and VLM-driven state updates.
An agent framework for structured long-form video understanding that integrates unified temporal and object memory with -ugmeentdd reasoning, enabling multi- round chain-of- thothtttht inference across complex video content.
Aigeo [
Across all these baselines, ReAgent-V almost outperforms in terms of accuracy, interpretability, and computational efficiency, largely due to its entropy entropy-guided guided frame selection and multi multi-pprspective reflection mechanism that support real-time reward-driven answer refinement.
Experiments are run on two H100-96GB GPUs with NVLink, using 64 frames per video. CLIP (ViT-L/14-336) is used for vision-language alignment. Videos are processed using decord and ffmpeg, with audio chunked for transcription. At inference, ReAgent-V performs tool-augmented reasoning by dynamically invoking OCR, ASR, DET and other modules. A multi-agent reflection stratey-ccomprisin conservative, neutral, and aggressive evaluators-is followed by meta-agent decision step.
A.1.2 Video LLM Reasoning
To investigate how ReAgent-V V can enhance the reasoning capability of Video LLMs, follow the experimental setup from [10) and apply ReAgent-V to filter the video portion of the Aideo-R1-2600 dataset. Specifically, during inference, we retain samples with importance scores lower than (out of 10) as these tend to be more challenging and thus more valuable for training. We then fine-tune the Qwen2.5- model 8 NVIDIA H100 80GB GPUs using the filtered dataset and compare against several baseline models defined the paper. Our results show that the model achieves 2.1% improvement in overall performance while using only 45% of the original training data, demonstrating the effectiveness of high-quality sample selection in enhancing video reasoning.
Benchmarks. We evaluate our model on six video benchmarks: VSI-Bench 12 VideoMMMU 13
MMVU [58] MVBench (22]. TempCompass [27], and VideoMME [.]. The first three benchmarks focus primarily video reasoning tasks that assess the model's 's ability to understand and reason complex video semantics, while the latter three are general video understanding benchmarks involving mix of perception and reasoning challenges. For MMVU, evaluate using its multiple-choice question subset to ensure stability and consistency.
A.1.3 VLA Alignment
To construct a preference- beeeeeence-based dataset for aligning lannonnllnnagg-ctttinn (VLA) models, we collect and process rollouts from the SIMPLER environment (23). Specifically, we construct reward-ranked dataset for VLA alignment by sampling trajectories from four original in-domain tasks using the Open
baseline released by [57]. with five rollouts per task. The remaining three generalization tasks are kept unseen and excluded from alignment. After data collection, use ReAgent-V to evaluate each trajectory segment, assigning scalar reward scores based on task success, stability, completeness, and accuracy. For each task, we sample 20 best-vs- worst trajectory pairs, matched by task type and initial environment state. This yields 80 total trajectories, which are then partitioned into two datasets: chosen set containing hiward trajectories and rejected set with low-reward counterparts.
(OpenVLAA-SFT) model, supervised-finetune
We conduct all VLA alignment training on single NVIDIA A100 GPU (80GB). The model is initialized from Open VLA-SFT.. Training employs LoRA (rank=32) with learning rate of 2e-5 and dropout of 0.0. We use gradient accumulation with step size of 4 to simulate larger batch sizes under
22

---

limited GPU memory. After training, the base model is merged with LoRA weights to obtain the final model. Our alignment process follows the Trajectory-wis Preference Optimization (TPO) [57] paradigm, where the objective is to learn reward-aligned policy using the preference pairs described above. All results reported in the main paper are obtained after 6,800 steps of alignment training.
We evaluate all VLA models in the SIMPLER environment, covering four types of generalization: in-domain, subject generalization, physical generalization, and semantic generalization. In-domain consists of the four original tasks in the SIMPLER environment. Subject generalization includes three new tasks involving objects not seen during training. Physical generalization comprises eight tasks created by varying the width, height, and size of known objects. Semantic generalization includes four tasks where the original prompts are rephrased with synonymous instructions. Each taskset comprises several subtasks, and each subtask is evaluated over fixed number of distinct episodes. The final metric is the average task success rate across episodes. For all baseline models Open-Oeen.V.. VI.ASTT,,, Open OpenVIA VLA-DPO, and Open VLA-TPO with GRAPE-we use results reported by [77. Our evaluation strictly follows the same experimental setup, adopting identical task definitions and environmental configurations to ensure fair and consistent comparison.
B
Evaluation Criteria and Prompts
B.1
Tool Selection Process
Tool Selection Prompt
[Task]
Carefully analyze the video content and identify exactly what information needs to be retrieved to support answering the given question. To answer the question step by step, list all the physical entities related to the question you want to retrieve, you provide your retrieve request to assist you by the following JSON format: [Tool Functions]
The Tool Factory supports the following types of retrieval via specialized tools:
Text Extraction: Tools: OCR, ASR extract embedded text (e.g., signs, timestamps) and transcribe speech or narration.
Object Detection and Grounding: Tool: Grounding DINO detects objects and
aligns them with natural language prompts.
Multimodal Matching: Tools: CLIP perform image-text alignment and retrieval
based on semantic similarity.
Scene Graph, Action Detector con- struct structured graphs of object relations and detect visual actions.
Structured Scene Understanding: Tools:
Video Reasoning: Tools: ShareGPT4Video, VQA Model enable video-based ques-
tion answering and explanation generation.
Caption Generation: Tool: Captioning Model generate scene-level textual de-
scriptions.
Emotion and Identity Recognition:
Tools: Face decognition. Emotion
Detector detect character identities and their emotional states.
[Output format]:
Use the following format to list the functional categories and corresponding tools required:
[
("function": "Function Category",
"tools": ["Tool1", "Tool2"]}
]
Example 1:
Original question: "How many blue balloons are over the long table in the middle of the room at the end of this video?"
Options: "A. 1. 2. C. 4.
Output:
23

---

("function"; "Visual Object Detection", "tools": ["Grounding DINO"]},
{"function": "Structured Scene Understanding", "tools": ["Scene Graph"]},
{"function"; "Numerical Reasoning",
"tools": ["ShareGPT4Video"]]
Example 2:
Original question: "In the lower left corner of the video, what color the woman wearing on the right side of the man in black clothes?" Options: "A. Blue. B. White. Red. Yellow." Output:
[
("function"; "Visual Object Detection", "tools": ["Grounding DINO"]}, ("function"; "Structured Scene Understanding", "tools": ["Scene Graph"]},
("function"; "Global Scene Description",
"tools": ["Captioning MJ]el"]}
Example 3:
Original question: "In which country is the comedy featured in the video recognized world- wide?"
Options: "A. China. B. UK. C. Germany. D. United States."
Output:
[
("function": "Audio Text Extraction",
"tools": ["ASR"]},
("function": "Global Semantic Summary",
"tools": ["Captioning Model"]}, ("function": "Commonsense Reasoning", "tools": ["ShareGPT4Video"]]
Example 4:
Original question: "Describe what the chef does to prepare the pasta dish in the video." Options: an
Output:
[
("function"; "Audio Instruction Extraction",
"tools": ["ASR"]},
("function"; "Object and Action Detection", "tools": ["Grounding DINO", "Action Detector"]], ("function"; "Structured Scene Understanding", "tools": ["Scene Graph"]},
("function": "Narrative Generation", "tools": ["Captioning Model"]}
[Now begin]
Note that you don't 't need to answer the question in this step, you don't 't need any infomation about the video image. You only need to provide your retrieve request (it's optional), and will help you retrieve the infomation you want. Please provide the json format.
This section introduces the Tool Selection Prompt framework, designed to guide awtre video question answering through structured tool invocation. By decomposing user question into

---

functional information needs such object detection, text extraction, or reasoning the prompt enables precise specification of which tools (c.g. OCR, Grounding DINO [26], CLIP, Scene Graph. ASR) should be employed. The unified JSON output format supports downstream automation and tool chaining, ensuring modularity and extensibility. Multiple illustrative examples demonstrate how diverse question types can be mapped to appropriate retrieval functions and tools, from counting objects to recognizing emotional states or extracting narrated instructions. Notably, additional visual tools other modalities are required, users can simply modify the tool selection section to include the necessary tools, making this prompt highly flexible and scalable interface for building robust multimodal QA pipelines.
B.2 Reflection Prompt
The Reflection Prompt framework divided into three complementary strategies Neutral B.2. Conservative B.2.2 and Aggressive[B.2. B.2.3 each offering different level of intervention revise initial video QA outputs. These prompts are designed support multi-stage reasoning and grounded visual correction by reassessing object perception, answer validity, full reasoning chains. De- pending on the task format (e.g.. open-ended QA, multiple choice), scoring mechanism (c.g., scalar reward, structured feedback), desired reflection granularity, users can select the appropriate strat- egy: Neutral for perceptual correction, Conservative for mability-focussee validation, and Aggressive for complete answer reconstruction. These templates also be adapted hybridized to support diverse evaluation workflows, enabling flexible deployment across multi-task QA pipelines.
B.2.1 Neutral Reflection Prompt
Entity-Centric Revision
[Task]]
You are neutral agent responsible for reassessing the initial model answer by correcting only visual misperceptions of scene elements. Your role is to revise the perceptual input (i.e., object/entity grounding) while preserving the original reasoning logic. Do not introduce any new reasoning steps.
[Reflection Requirement]
If the original answer is based on misidentified visual entity, correct that grounding (e.g., object type, color, spatial position). Keep the interpretation process unchanged. This strategy focuses on refining what is seen, not how it is reasoned. [Reflection Procedure]
Re-examine the video or image frames for object-level details (e.g., people, objects, colors, gestures).
Determine whether the initial answer failed due to incorrect or missing perception of visual entities.
3. Adjust the relevant scene elements accordingly (e.g., update object color, position, or identity).
4. Reuse the original reasoning chain, now applied the corrected visual grounding.
[Evaluation Guidelines]
Remain neutral - do not assume the initial answer is incorrect unless there is clear evidence of perceptual error.
Do not modify the reasoning logic preserve the original inference path.
Only revise the interpretation of visual content (i.e., what objects appear and their attributes).
[Input]
Question: {text)
Initial Answer:
Eval Report: (structured feedback or scalar reward)
25

---

[Output]
Use the following format to provide the final revised answer after entity-level correction. Only output the revised answer and its confidence score no explanations, no justifications, and no extra text of any kind.
Final Answer:
(ffree-form revised answer), confidence score: 0-1)
B.2.2 Conservative Reflection Prompt
Answer-Focused Revision
[Task]
You a conservative agent responsible for validating the initial model answer. Your role is to preserve the original answer unless there is irrefutable visual evidence that directly contradicts it. Do not revise reinterpret the reasoning scene elements unless the contradiction is absolute.
[Reflection Requirement]
Only revise the final answer itself-de do not modify scene elements or reasoning logic. If the original answer remains potentially valid, it should be retained. This strategy aims for minimal intervention: conservative reflection focuses on maintaining output stability unless overwhelming visual contradiction present. [Reflection Procedure]
1. Re-examine the visual content for any direct contradiction to the initial answer.
Accept only literal, unambiguous visual cues that fully invalidate the original answer.
Do not alter object interpretation, scene structure, or logical flow.
4. Revise the final output only if the contradiction is undeniable and renders the original answer unsupportable.
[Evaluation Guidelines]
Retain the original answer if:
Any uncertainty ambiguity exists in the evidence
Visual information lacks a clear, literal contradiction
A reasonable observer could still accept the original answer
Revise the answer only if:
The visual content presents overwhelming and explicit contradiction
The revised answer exactly matches visible evidence without requiring inter- pretation
The contradiction is strong enough to convince any neutral evaluator
[Input]
Question: {text)
Initial Answer: (initial_answer
Eval Report: (structured feedback scalar reward)
[Output Format]
Use the following format to provide the final decision after conservative validation. Only output the revised answer and your confidence score explanations, justifications, and no extra text of any kind.
Final Answer: (ffree-form revised answer), confidence score: 0-1)
26

---

B.2.3 Aggressive Reflection Prompt
Deiiinn Reasoning- Driven Revision
[Task]
You are an aggressive agent responsible for actively challenging the initial model answer. Your role is revise both the reasoning process and the visual understanding in order to reconstruct a superior alternative. [Reflection Requirement]
This strategy requires modifying both the reasoning steps and the associated scene entities. It involves the widest scope of change and is intended completely overturn the original logic and rebuild more accurate answer from scratch. Accept loose semantic alignment, reinterpret ambiguous scenes, and prioritize alternative perspectives over the original. [Reflection Procedure]
Re-examine all video frames for cues that could support a different interpretation. Modify the understanding of relevant visual entities (objects, attributes, spatial relations).
Reconstruct the reasoning chain based on the newly grounded observations.
Select new answer that best fits the revised reasoning and evidence.
[Evaluation Guidelines]
Always replace the original answer the default assumption is that it is suboptimal or incorrect.
Consider alternative answers even when based on partial, ambiguous, or abstract cues.
Allow semantic flexibility (e.g., "canine" = "dog", "liquid" = "wattrr") and recon- textualization.
Prioritize comprehensive reinterpretation to generate improved final response. [Input]
Question: {text}
Initial Answer: Sinitial_answer
Eval Report: (structured feedback or scalar reward]
[Output Format]
Use the following format to provide the final revised answer after full reasoning and entity- level revision. Only output the revised answer and your confidence no justifications, and no extra text of any kind.
explanations,
Final Answer: (ffree-form revised answer), confidence 0-1)
B.2.4 Overall Reflection Prompt
Multi-Perspeetive Reflection Aggregation
[Task]]
You specialized Meta-Agent for video question answering. Your role is integrate the answers and confidence scores from three agents with different reflection strategies. Your goal to synthesize final answer by evaluating answer quality, confidence levels, and semantic overlap.
[Multi-Perspective Inputs]
Conservative Agent (Answer-Focused Reflection)
Answer: conasverrccnssrrativve... Confidence: (conf_cooservattive)
Neutral Agent (Entity-Centric Reflection)
Answer: Sanser_neutraal Confidence:
27

---

Aggressive Agent (Reasoning- -Divven Reflection)
Answer: Bansser_aggressive), Confidence: Sconf_aggressivee
[Decisoon Procedure]
Step 1 HighhCCofiidenc Fusion
If all three confidence scores exceed their respective thresholds:
consesvvveev 2 0.6
conf conf_neutral 0.7
conf confaggrressvee 2 0.8
Then:
Combine the three answers Sansver_aggrestte (ansver_aggressiie (answer_aggressive))
Extract shared components and consistent semantic information
Remove contradictions or unsupported segments
Produce final, coherent free-form answer that integrates the common insights
Step 2 - Confidence-Basse Selection one or more confidence scores fail to meet their thresholds:
Select the answer with the highest confidence score among the three agents
Use that agent's full revised answer as the final output
[Evaluation Criteria]
Semantic Overlap: Identify key phrases, facts, and themes that appear in multiple answers
Contradiction Removal: Discard any segments that directly conflict with others
Fluency: Ensure the final answer reads as a natural, well-formed sentence [Input]
Question: {text}
Initial Answer: Sinitial_answey
Agent Answers:
danswer__onservativee
Yanswer_neutral)
(answer_aggresst)
Agent Confidences:
{conf conneestveeevvivee
{conf_aggressiive]
[Output Format]
Use the following format provide the final revised answer. Only output the revised answer no explanations, no justifications, and no extra text of any kind.
Final Answer: (ffree-form revised answer)
B.3
Critical Prompt
This section introduces two complementary evaluation prompts Clarification Question Genera- tion B.4]aad Eval Report Generation designed to support critical diagnosis of model answers in video question answering. The Clarficationn Prompt generates precise, targeted sub-questions when an answer is incomplete or ambiguous, uncovering reasoning or grounding gaps by refer- encing specific visual or contextual elements. It is particularly useful for both open-ended and multiple-choice QA tasks, helping localize errors without requiring immediate scoring. The Eval Report Prompt provides a structured scoring mechanism across five dimensions-vvsus alignment,
28

---

temporal accuracy. option disambiguation, reasoning specificity, and linguistic precision mrelsion--yellding both qualitative feedback and scalar reward useful for leaderboard tracking. model tuning, or reinforcement learning. Together, these prompts enable flexible and task-adaptable evaluation framework: clarification questions are ideal for diagnosing failures and guiding revisions, while scalar scores support performance benchmarking. Depending on the evaluation goal-erro localization, answer justification, or progress tracking-thee -thes prompts can be selectively applied or adapted, with clarification prioritized in open-domain QA and scoring emphasized in competitive or quantitative settings.
B.4 Critical Question Prompt
Clarification Questions Generation Prompt
[Task]
You are critic agent tasked with evaluating the quality of the initial answer Ag generated by the target agent, based on the provided question, context, and video content. If the answer is deemed unsatisfactory, your goal is help localize potential errors by generating more sub-questions. These sub-question. mustshould be highly specific and firmly grounded in the visual contextual evidence.
[Input Data]
Question: "(text)"
Answer: "[answer}"
Context: "[context]"
[Evaluation Criteria]
Check the answer fully addresses the question
Verify key elements from context are included Assess whether video content supports the answer
4. If not, raise sub-questions to expose missing uncertain reasoning
[Clarification Guidelines]
If the answer is incomplete, generate 1-3 ultra-specific clarification questions following these rules:
Must start with: "What", "Where", "When", "Which", or "How"
Must reference concrete elements from context/video
No vague pronouns ("it", "they") use specific nouns
Examples:
"Which timestamp shows the error?" "How many frames were processed?"
[Output Format]
Return
(for complete answers)
Return ["questioni?", "question2?", ...] (for incomplete answers)

---

B.5 Evaluation Report Prompt
Eval Report Generation
[Task]
You are critic agent tasked with evaluating the quality of the initial answer Ao generated by the target agent, using the given question and contextual information. Your goal to provide structured diagnostic feedback by scoring the answer across multiple dimensions and computing final scalar reward (0.0-1.0) based on the total score.
[Input Data]
Question: {text}
Context: {context}
Initial Answer: Simitial_answent
[Evaluation Criteria]
Rate the answer on the following five dimensions (0.0-5.0 scale for each):
Visual Alignment: Is the answer aligned with visible video evidence?
Temporal Accuracy:
answer consistent with the timeline or timestamps?
Option Disambiguation: If multiple options are similar, does the answer clearly
justify the selected one?
Reasoning Specificity: the reasoning clear, focused, and appropriately detailed?
Linguistic Precision:
Is the answer grammatically correct and semantically accu-
rate?
[Output Format]
Return JSON object with detailed scoring and reasons for each dimension, plus the final total and normalized scalar reward:
{
"scores": {
"risual_alignment"/ {"value": float (0.0-5.0) "reason": "..."),
atemporal_accuracy"" ("value": float, "reason": "...",,
disambiguattinn ("value": float, "reason": "...") "}, "reasoning
'reasoming_specifcccii specificity"; f"value": float, "reason": "..."}
ciaguusstic_recisioonn ("value": float, "reason": "...")
"total_score" float (0.0-25.0),
B.6 Implementation Workflow of ReAgent-V
Figure 6 illustrates the complete inference workflow of ReAgent-VV V, modular agent system de- signed for video-based question answering. The pipeline begins with unified initialization via load_ load_defaal default, followed by ECRS-based keyframe selection to reduce redundancy while preserving semantic relevance. A dictionary-based tool selection mechanism dynamically activates symbolic extraction tools (e.g., OCR, ASR. DET) based on the input query. Extracted textual context is merged into modal_ modal_strings and composed into a multimodal prompt for LLaVA-based initial answering. If critical gaps are identified, the system enters a reflective reasoning stage to revise the answer. Finally, an evaluation report is generated assess the quality of both the initial and refined responses.
C More Visualization Results
30

---

ReAgent-V Inference Pipeline
>>> from init_modules import
ga_system ReAgentV. loaddlddd_a ("ftaull"ppath/tt//baeeee. _moeel))
frames qa_eystem. rideo_framee ("example .pp"", max_ faa_es
>>>
key_ key_frames, key_indices = ma_system.ECCe aeres_eeeeet (frames, question)
>>>
Pass custom tool list dynamically control which tools apply and revise the tool selection prompt template accordingly.
>>>
tool_list ["OCR", "ASR", "DET", "SceneGraph", "Grounding DINO", Caption Model", "CLIP", "ShareGPTaVideo", "VQA Model", "Action Detector", "Face Recognition", "Emotion Detector"]
"selected_ tools"" is boolean map indicating required tools for answering the question, e.g., {"OCR": True, "ASR"; False, "DET": False, ...}
>>> tools qa_aystem. weweectetoleleseeeeeeeeeeee
tool_listttoolllsst
>>> Use selected_tools to only necessary models and extract tool specific information from key_frames into modal_info, a dictionary keyed by tool name.
>>> modal_infe info = aystem. extract_modal info (key question, **
(elected_tools)
Build multimodal prompt by integrating tool tool-apecific information from modal_info into the question template.
>>> prompt qa_system. hasild__ulimmmda_ppp (questioon, codal_info, joy_indices, len (kee_ffrmms))
>>> initial_ answer, qa_aystem. q8_eystem.model wwdemmoodeeeeeeeeeeeeeeeeeeeeeee ieeelliifeeeeeeeee
(prompt,, (prompt, key_frames)
Critic-driven refinement if necessary
>>> critical_qe
pesstem.enerate... aaritical_questions insteal modal_ing info, key_frames)
(qusttionsuuuustiin... (question,
>>> if critical_qs:
{q_i: {tool_name: info)}, mapping each critic question tool-specific info. for q.1 in critical.qs:
updated_infor
tools_i tes_eyyttm.eetratt_mmo extract_modas info (key_ frames, q-i, **tools_i) info_i = qa_system...
seysstemselect_tone.
tool lilllisstttollisss
padated_infos(q.3
info_i
Wrap the original modal_inft into {question: modal_ info) contert_inoos = {question: modal_ info, )nnfos # Merge all into unified context dict
report qa_system.... Sea_syyyemggeerattee..
mitiial_nsseer
context_ key_ (ramess
ltt_reeleetive_iaaal__oomm applies three reflection strategies ( neutral, aggressive, conservative) and merges the best answer using the weerall_prompt_template
simal_answer qa__ystem.get.c ret_refleettie_finaal. question, initial_answer report, key_ frames)
>>> else:
Sinal_answer initial_answer
Figure 6: ReAgent-V inference pipeline: after ECRS keyframe selection, tools are dynamically selected generate initial answer. If critical questions arise, tool outputs updated for reflection. Three reasoning strategies are used to revise or confirm the answer.
C.1 Visualization of Frame Selection
These case studies qualitatively demonstrate the effectiveness ECRS frame selection compared to uniform sampling across range of video question answering scenarios. In each example, FCRS consistently captures frames that are more semantically aligned with the question, providing richer context and clearer evidence to support reasoning. For instance, in Figure 7 ECRS selects
31

---

Four Frames Sampled Using Uniform Sampling
5 ONE
57
Four Frames Sampled Using ECRS Sampling
Question about the video How did the photographer who took pictures of the three tourists in the video take the photo?
Figure 7: ECRS sampling better captures the interaction between the photographer and the tourists.
Four Frames Sampled Using Uniform Sampling
The UNARIOUT
Four Frames Sampled Using ECRS Sampling
Question about the video What organ did the woman in the video remove from the medical model?
Figure 8: ECRS highlights the woman's action of removing the organ more clearly than uniform sampling.
frames that highlight the interaction between the photographer and tourists, directly addressing the question about how the photo was taken unlike uniform sampling, which includes generic landscape shots. In Figure ECRS captures the precise moment when the woman interacts with the medical model, effectively grounding the answer to the question about organ removal. Similarly, Figures 9 and 1O that ECRS preserves critical moments involving text and actions-suuc the student explaining his motivation for eating the banana or the banana-taping act that contextualizes the replaced painting-while uniform sampling misses misaligns with these moments. Across all examples, ECRS provides temporally and semantically focused evidence that improves alignment between selected frames and the target question, validating its superiority in supporting visual reasoning. Note that although ECRS typically selects 20-40 informative frames per video for downstream processing, only four representative frames are visualized here by evenly sampling from the selected set, in order to maintain clarity and consistency in comparison.
C.2 Visualization of Evaluation Report
These four evaluation report cases demonstrate how ReAgent-V leverages visual tools and multi- -ggett collaboration to refine or correct initial answers by grounding them more accurately in visual and textual evidence. In Figure [1] the system correctly identifies the Dragon Boat Festival as the main theme of the video by combining OCR and DET outputs to interpret cultural symbols and scene elements. Figure 12 shows quantitative reasoning adjustment, where the agent detects two birds and
32

---

Four Frames Sampled Using Uniform Sampling
rnnakent aft hamara that Itakian artir MartidaCaliiaa
Nehittgn sco becisse vis Tungyy
Four Frames Sampled Using ECRS Sampling
ling
sco
wall
becausee herwas Tunggy:
wwsk of crddem aooblab aebbaar
Question about the video Based the video, which of the following descriptions the reason why the student ate the banana?
Figure ECRS sampling emphasizes the reason behind the student eating the banana with supporting text.
Four Frames Sampled Using Uniform Sampling
Four Frames Sampled Using ECRS Sampling
Question about the video Which elements are depicted in the painting introduced by the video?
Figure 10: ECRS captures the banana-taping act that replaced the shredded painting, providing better context.
revises the initial bird count from three to two, aligning the answer with grounded visual analysis. In Figure [3] the model originally misidentifies the cat's 's color, but after grounding the detected regions and reviewing visual attributes, the agents collaboratively revise the answer the correct "orange and white." Finally, Figure [4 illlustratess a geographic correction where the system initially mislabels the terrain "tropical," but after integrating CL.IP-based semantics and detailed grounding, it updates the answer to "polar," aligning better with snowy and icy visual cues. These examples highlight how reflective critique and grounded evidence improve factual accuracy and visual-textur alignment.
C.3 Visualization of VLA Alignment
Figure 15 visualizes six robotic manipulation tasks and highlights the progression from failure success after applying iidectioo-guideds reward correction during policy fine-tuning. In each carkot placement, spoon placement, eggplant basket placement, cube stacking, coke can placement, and sprite can placement- the left sequence illustrates robot's initial failure to complete the action correctly, while the right sequence demonstrates the corrected behavior after fine-tuning. The visualizations show that the robot learns adapt positioning, trajectory, and precision based on reflective feedback. For example, in the carrot and spoon placement tasks, the
33

---

Question
Which traditional Chinese festival is primarily showcased in this video?
A. Dragon Boat Festival. В. Spring Festival.
C. Mid-Autumn Festival. D. Double Ninth Festival
To extract any Vis lble Chinsse charactera that explidilyy reference festival
To detect key trttonal objects(e g.. food, herba, ussociatad specific festhrals.
Evaluation report
Structured Feadback:
in answer Dragon Festival accurately rellects the visuel and textual euicence in the viceo. Early frames showzongg,, dmagon boat imagery, and catamus... all strongly fied this OGR identified characters like "N offering clear Uigulattic support. confimmed symbolio items such as peddles and leat-wrapped foods... tracitionilyy linked to the occasion. Wh elements like lew sela bennere appear other festivals, the consistent presence Dragon Bost-seeciiic visuela acress scencs cantima answer. well supported both semantically and visually.
EXT
OCR
DET
festwal..
Target Agent
Initial answer 1 A
TV
Scale Reward:
Alighmmnt:: 4.5/5.
Tempomi Accuracy:
Critic Agent
Option Disemtipuution: 4.D/5. rewsent
Reseoning Specifclity 4.5/5.0, O, resson:
Linguistie Precision: 4 4.055.0, reason...
Total Score: 21.0/25.0
Visual and textual cues confirm that the video primarily showcases the Dragon Boat Festival.
Figure 11:
Question
How many birds are there in the video??
C. 3. D. 4.
Conservative
Evaluation report
Cortdennec 06
locore ner matchinthhe prunet tars..
Structured Feedback:
The init snaove & Pix nasecrable
glace basse un cur dctuntionss Boors
Gruch izunti snt two hird-ake ctpucts, stsible
Reeurral
Grounding DINO
Hiowswer, 1z partatlyl occluded and bluured in the uppor froma--e na riloout fiv inter contimmed rugh refined groundiig and adiono).. The unwuoouuuuu stons fromiiiited dotection granularity andooettappengg. tralansess Su whitic 2. conttullyy sutpprted, cummmttttyy
Confidiice 2.
Target Agent
Initial answer: B
hare partiall hindee mlinngs and
out tichane betweaoe theaarott?
sperial mestiinehips n the scono
2. Courd close prtodmi by mantaper nal hreee cousod nwo alds popoar as ono? Donew setewbrdb-aaoo appoa oo progresses?
Aggreasvee
Multi-Agant Evaluntion
Add Tool:
DET
Scale Reward:
Vince Aiggment 3.1/5.8, reason Torpee Arourata A n masser
Reflection
Option D' Dloanto guartone DOLD menwors
Reasoning Scecifilte 2.2/5 Ling, ladi Predision 1. 3.8/5.0. resson.... Total Scars: 18.5/22.
Final answer'
Critie Agent
Agent
Figure 12: Multimodal analysis revises the bird count in the video from two three after detecting an occluded bird.
gripper initially misaligns with the target location but is corrected center the object on the plate or towel. In the eggplant and cube stacking tasks, the robot improves its grasp and drop accuracy, successfully placing the object in or on the intended target. In the coke and sprite can tasks, the robot adjusts its vertical alignment and release timing ensure stable placement. These results collectively demonstrate that reflection-driven fine-tuning enhances the robot's task completion reliability across diverse object manipulation scenarios.
34

---

Question
What colar is the cat in the video? Orange and white. Orange.
B. Black and white.
Conservative
Cookdbree
Exalmation report
memdddheecti armeer cherce win the viuual vieii
To gororatt scone lovel denorgption like "An crange cal bying on bad.""
Structured Foodbook:
The intie annoer c. Orange a INsecracin gunaa bused high
underalanding Captioii
Neutral
CLIP
Caption Model
derecct and lncolize cor
favored
Ht senvo, Uhe utose white patchees on the underside and pawas deea
Colfereenc
Targef Agent
more debated
partist both tools... inading appearanio.
Initial
2. the cufoo actually coloorredi osen the comtinnt cnleris orange?
Qucctiomn
beily
metmay inscho
mumtile visuet
taate
manaepttion owertanting hooster nature. wiike C. Orangeis plausisle I''s nosmpllte glven the vieual denice.
Aggressivve
Mutti-Agant Evoluation
Scate Hewardd:
Grounding DINO
AiscelAlarments
Add Tool:
Sprizn LS guoiore D. nosort
Accursoy 5.5
Reflection
Reassringg Specifiiity 3.800 resson... Urguatil Preciaion 4.055. reasore...
Final answer
Critic Agent
Targel Agent
Figure 13: Multi-agent collaboration corrects the cat's color in the video from "Orange" "Orange and white."
Question
Which regiappni padeee are primarily festured this video? Tropical Terperate zegions.
Polar regions. Subtropical regions.
Canttimnn 1esile cop.
?
Evaluation report
moroh dedianntrrseeeeeeeee choice u visual
Seructured Fondback
Tropinet reg based blus eatern in early Bramee. CL liely el gind these wntooookiinngggnn udd frenco coebengo.
content of the vibeo.
descriptio and termin tyoe.
Neutral
CLIP
Caption Model
Contdence 3.7
Targel Agent
peace. guacliwwe
conner
missee
tnitial
Gritl Quastions
Do any framss suggest colder climatia, th evidence widdinn, sermin, uolor typiealy usooiatedd thith othrr reglanu?
earmetta loe Tosteraa
ratural
toots.. These no shift soone soont stcuity loverd potes reahorn.
Thane is plaue partal
Agaressive
Muttl-Agent Evaluation
Scate Roward:
Grounding DINO
Alggment: rerasarr
Add Tool:
Disombrguudd
Reflectoon
Reesoning Specificcle22 2 meont....
Ungullttc Prectsion resso Boor 11.6/25.
Final
Critic Awent
Taruel Aweel
Figure 14: Agents identify icebergs and snowy terrain, correcting the geographical region from "Tropical" to "Polar."
D Additional Experimental Results
D.1 Trends of Selection Scores Across Videos
Figure 16 illustrates the selection behavior of ECRS across various videos by comparing its score distribution with CLIP and Entropy- based baselines. The red bars indicate that ECRS consistently selects frames near local peaks, capturing semantically or visually informative moments. Compared to CLIP and Entropy score selections, ECRS demonstrates more temporally clustered choices that
35

---

Tisk: put the carror the plate (Saied siccess)
spocn suoces)
(a) Carrot placement
(b) Spoon placement
pui spgplent booduct
Taok: sockk green cube the yellow (friled sucoos))
(c) Eggplant basket placement
(d) Cube stacking
Task: pur the wake the toeel (lailed success)
ttalled
(e) Coke can placement
(f) Sprite can placement
Figure 15: Visualization of robotic task executions before and after policy fine-tuning using reflection- guided reward correction, showing progression from failure to success across four tasks: (a) carrot placement, (b) spoon placement, (c) eggplant basket placement, (d) cube stacking, (e) coke can placement, and (f) sprite can placement.
align with high- tigh-coomaattess regions, indicating its ability to perform more focused and discriminative frame selection throughout the iterations.
D.2 Frame Index Trends Across Iterations
Figure 17 presents the iterative frame reduction behavior of ECRS across multiple videos, showing trends average cofpaatppsss (blue), GT-compactness (orange), and the number of selected frames (green). Self-compactness refers to the average index distance among the selected frame indices each iteration, indicating how temporally clustered the selections are. GT-compactness measures the average index distance between selected frames and ground truth-relevant frames, reflecting how well the selected subset aligns with semantically important moments in the video. As iterations progress, both compactness metrics consistently decrease, demonstrating that ECRS not only reduces the number of selected frames but also improves their temporal tightness and alignment with ground truth. The number of selected frames drops sharply in early iterations and then stabilizes. This trend highlights ECRS's ability efficiently eliminate redundant frames while preserving compact, semantically meaningful subset aligned with human- annotated content.
36

---

1.00
Normelizdd Score 2005-20
0.8
Normalized
Cita Scoee
Seare
Morealll de
Erirpp Scone
ecns Host Ronosedd
0.2
RCAS taust Rerptie)
Frame Index
30
50
60
Frome Index
Frame Index
30
00
Normelized Scare
Normalize 5
Overden
Encrey Scor
Normelized
EORS
ECRS
U.
0.4
60
50
00
20
20
40
50
00
Frame Index
Frame Indexx
Frame Index
Hormalized Scort
1.00
Sectie 6-885
1.
Ci
Normalized Score
1.0)
Secalee
0.6
Clpy
CCRS
Normalized
Se
RORS
0.6
ECRS
0.
0.
30 Frama
0.0"
30
Indax
Nomated
8
Cio Score
Tresma Irdee
Figure 16: Comparison of frame selection strategies across multiple videos, illustrating how ECRS (final iteration), Entropy Score, and CLIP Score vary with frame index and influence selected keyframes (highlighted in red), along with their overlap with ground truth frames manually annotated by experts refer to the VideoMME dataset for corresponding video IDs and validation details.
denge 500
Mengee 1sta
aeinge 1500
Wesg D151
-th Jestage Esttare PiCE
Frccpp D 177L
Ancrpe Eitoos EiC Dongstuaaco
Biraliee
Derallas
Drarika
Breddion
Na Santection
This figure shows tracks two compactness metrics- self-compactness (left y-axis, mean squared distance to selected frames' centroid) and GT-compactness (left y-axis:, mean squared distance to selected and ground truth centroids) and the number of selected frames) along with frame count (right y-axis) across iterations (x-axis) for videos from the (Video-MME/MLLUT datasets. L), where lower left-axis values indicate tighter clustering.
Figure 17:
D.3 Comparison of Frame Selection Methods Across Varying Input Frame Numbers
Figure [8] shows that ECRS consistently outperforms other frame selection strategies across all frame counts on the VideoMME dataset. As the number of input frames increases from 8 to 64, accuracy improves across all methods, and the performance gap between strategies remains consistent. While these trends are observed across different base models, the results shown here use ILAVA-Video-77B which exhibits particularly strong performance, further highlighting the effectiveness of ECRS in enhancing model accuracy through more informative frame selection.
37

---

Frame Selection Comparison VideoMME Dataset
CLP Score
Entropy
ECRS
Uniform
Accuracy (96
57
55
8
16
32
64
Number of Frames
Figure 18: Frame selection strategies on VideoMME (LLaVA- V--VViido--7B).
Table 5: Ablation study the effect of Visual Tools across different base models and benchmarks.
Model
Visual Tools
LongBench
EgoSchema
VideoMME
53.1
60.8
57.9
LLaVA-Video
46.1
56.7
56.3
65.3
74.0
75,5
LLa LaVo--iideo-22B
60.9
75.0
67.3
46.4
56.4
58.3
Qwen2-VL-7B
44.9
55.6
53.8
66.4
76.2
75.1
Qwen2.5-VL-72B
62.6
75.7
74.3
D.4 Ablation of Visual Tools
Table 5 presents an ablation study on the impact of incorporating Visual Tools across different base models and benchmarks. Across all settings, enabling visual tools consistently improves performance on LongBench, EgoSchema, and VideoMME. The gains are particularly notable for LLaVA- LLLVA-Video--- and Qwen2-VL-7D where enabling visual tools leads substantial improvements-forr instance, +7.0 on LongBench and +4.5 on VideoMME for LLaVA-Video- smaller for larger models like -aaV--Vddo--22B and Qwen2.5-VL-72B., improvements are still evident on LongBench and VidcoMME. These results indicate that visual tools provide valuable auxiliary signals that enhance multimodal reasoning, especially for smaller or less capable base models.
While performance gap is
Limitation
Despite employing keyframe selection, handling long videos remains challenging due informa- tion redundancy and complex semantic relationships, which hinder the model's ability to achieve comprehensive and accurate understanding and reasoning. Additionally, the reflection and evaluation mechanisms rely on heuristic-based rules or templates, lacking adaptive and end-to-end learning capabilities, which may limit the effectiveness of automatic error correction.
Social impact
Our method enhances video understanding and reasoning, which can benefit education, assistive technologies, and human-computer interaction. However, potential risks such as privacy concerns and the propagation of dataset biases should be carefully considered during deployment.
38


Reviews:
[{"review_id": "ptdDdHbhTh", "reviewer": "Reviewer_Nwjt", "summary": "This paper proposes ReAgent-V, a reward-driven multi-agent framework for video understanding. The framework consists of three parts. First, it selects frames relevant to the query by an entropy-calibrated selection strategy. Second, it invokes various tools (e.g., text extraction, object detection) to assist the reasoning process. Finally, it introduces a multi-perspective reflection mechanism, where a critic agent produce suggestions to guide the target agent to refine the answer. The paper compare ReAgent-V to multiple video understanding baselines on multiple video tasks. The paper also quantitatively and qualitatively analyzes the contributions of each component", "questions": "1.Could the authors provide more quantitative results on the selection quality of ECRS? Are there failure modes where ECRS excludes or over-selects critical frames? \n\n2.Could the authors clarify how the conservative, neutral, and aggressive reflection strategies differ in practice by cases? How robust are these strategies to prompt during reflection?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n1.The proposed framework addresses the important limitation of static, single-pass video reasoning in large vision-language models. By integrating real-time reward generation and multi-perspective reflection, ReAgent-V enables dynamic, self-correcting, and tool-augmented inference, which is highly relevant for complex video understanding scenarios.\n2.The experiments are solid and comprehensive. The authors conduct extensive experiments on 12 datasets across three applications—video understanding, video LLM reasoning, and vision-language-action alignment. Results show clear performance gains over prior baselines. The paper also includes ablations for frame selection and reflection mechanisms, showing meaningful improvements in both accuracy and efficiency.\n3.The agent frame work is extensible to different backbone models.\n\nWeaknesses\n1.The paper lacks quantitative analysis on the accuracy of ECRS. There’s no data showing how well frames selected by ECRS align with actual keyframes related to the query.\n2.The paper uses prompts to control the behavior of the three different reflection agents (conservative, neutral, and aggressive), but it doesn’t provide quantitative evidence showing whether these agents truly exhibit distinct behavior patterns.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful feedback, and we respond to your points individually as follows:\n\n**Q1:** The paper lacks quantitative analysis on the accuracy of ECRS. There’s no data showing how well frames selected by ECRS align with actual keyframes related to the query. Could the authors provide more quantitative results on the selection quality of ECRS? \n\n**A1:** Thank you for your insightful question! To quantitatively assess how well ECRS-selected frames align with ground-truth keyframes, we randomly sampled 100 videos from the VideoMME dataset, LongBench dataset and manually annotated the keyframes for each. We then compared ECRS against two common baselines - CLIP-based similarity and Entropy-based selection - on both VideoMME and LongBench benchmarks using five evaluation metrics: Recall, Precision, IoU, F1 Score, and Redundancy (defined as 1 minus the average pairwise CLIP similarity, where lower is better). \n\nAs shown in Table R1, ECRS significantly outperforms both baselines across all metrics. On VideoMME, ECRS achieves markedly higher Recall (0.891), Precision (0.872), IoU (0.816), and F1 Score (0.623), while reducing Redundancy (0.182) compared to CLIP (0.763/0.649/0.607/0.486/0.242) and Entropy (0.604/0.570/0.455/0.421/0.279). On LongBench, which involves long-form videos with sparse but critical evidence, ECRS maintains top performance with Recall (0.852), Precision (0.841), and IoU (0.891), again outperforming CLIP and Entropy by large margins and demonstrating its robustness in both short and long video settings.\n\n---\n\n**Table R1:** Quantitative Evaluation of Frame Selection Accuracy\n| **Dataset** | **Method** | **Recall** | **Precision** | **IoU** | **F1 Score** | **Redundancy** |\n| ------------- | ---------- | ----------- | -------------- | --------- | ------------- | --------------- |\n| **VideoMME** | **ECRS** | **0.891** | **0.872** | **0.816** | **0.623** | **0.182** |\n|  |CLIP | 0.763 | 0.649 | 0.607 | 0.486 | 0.242 |\n|  |Entropy | 0.604 | 0.570 | 0.455 | 0.421 | 0.279 |\n| **LongBench** | **ECRS** | **0.852** | **0.841** | **0.891** | **0.601** | **0.196** |\n|  |CLIP | 0.621 | 0.618 | 0.379 | 0.468 | 0.257 |\n|  |Entropy | 0.463 | 0.339 | 0.312 | 0.427 | 0.287 |\n\n---\n\n**Q2:** Could the authors clarify how the conservative, neutral, and aggressive reflection strategies differ in practice by cases? How robust are these strategies to prompt during reflection?\n\n**A2:** We sincerely thank you again for this interesting question. The prompts for all three strategies are provided in Appendix B.2. Specifically, when the base model's answer is deemed unsatisfactory, the conservative strategy only modifies the final answer while leaving the rest of the response unchanged. The neutral strategy makes moderate adjustments, including correcting the answer and any relevant nouns. The aggressive strategy goes further by not only correcting the answer and related nouns but also modifying the logical relationships within the sentence. For an analysis of the robustness of each strategy, please refer to Figure 4 in the main text. We report both the probability of making edits and the post-editing accuracy for each strategy. The results show that while each individual strategy can improve the response to some extent, combining all three strategies yields the highest overall accuracy."}, {"review_id": "ClFX3SDAPy", "reviewer": "Reviewer_dAzj", "summary": "The paper proposes an agentic video understanding framework. It involves a frame selection to pick salient input frames. It also provides a tool factory to assist model for video understanding. To supervise the model, it designs a critic agent for evaluating model results and providing evaluation report for improvement guidance. The target agent then refine based on three strategies to produce better predictions.", "questions": "Please provide analysis for model robustness on hyper-parameters and discussion on the effect of stronger MLLM backbone.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The paper proposes a new framework for agentic video understanding, and shows the framework is versatile and can support both test-time improvement and training time data construction.\n- The proposed entropy-based frame selection is interesting and has demonstrated consistent improvement.\n- The paper writing is easy to follow with clear structure.\n- The experimental results are reported for three settings to verify method effectiveness.\n\nWeakness:\n- The selection threshold used in Eq(4) involves multiple parameters. It is unclear if the method is robust to those parameters or requires careful parameter tuning.\n- In Table 1, the best performance is achieved with Qwen 2.5 72B, which is a larger MLLM than those used in [8]. Results with similar scale of MLLM shows mixed performance. It makes it hard to understand the improvement from the MLLM vs improvement from the method.", "was_revised": true, "final_justification": "I appreciate the rebuttal from the authors. After reading it and the opinions of the reviewers, I will maintain my positive rating.", "rebuttal": "Thank you for your thoughtful and constructive comments, we address each of your concerns in detail below:\n\n**Q1:** The selection threshold used in Eq(4) involves multiple parameters. It is unclear if the method is robust to those parameters or requires careful parameter tuning.\n\n**A1:** We sincerely thank you for this thoughtful question! This is indeed an important aspect that we should have clarified in the main text. Due to time limitations, we only included partial results earlier. Here, we conduct a more comprehensive robustness study on the parameters used in Eq (4), specifically the scaling factor \\(k\\) and the threshold $\\tau$, using Qwen2-VL-7B as the base model on the short split of the VideoMME. We vary the scaling parameter $k \\in \\{1, 2, 3, 4\\}$ while keeping $\\tau = 0.7$, and observe the following results:\n\n**Table R1:** Robustness Analysis of Parameters in Eq(4) on VideoMME (Short Split, Qwen2-VL-7B)\n| \\(k\\) Value | Accuracy (%) |\n|------------|--------------|\n| 1          | 73.5         |\n| 2          | 71.5         |\n| 3          | 74.5         |\n| 4          | 72.8         |\n\nWe also fix $k = 1$ and vary the threshold $\\tau \\in \\{0.5, 0.6, 0.7, 0.8\\}$:\n| ($\\tau$) Value | Accuracy (%) |\n|---------------|--------------|\n| 0.5           | 71.8         |\n| 0.6           | 72.3         |\n| 0.7           | 73.5         |\n| 0.8           | 71.2         |\n\nOverall, the results show that our method is robust to the choice of parameters. Performance remains relatively stable across a wide range of $k$ and $\\tau$ values. We also note that setting $\\tau$ above 0.8 may result in failure to select enough frames (e.g., 32), as the threshold becomes overly strict. Based on these observations, we adopt default values of $k = 1$ and $\\tau = 0.7$, which provide strong and stable performance without requiring fine-grained tuning.\n\n---\n\n**Q2:** In Table 1, the best performance is achieved with Qwen 2.5 72B, which is a larger MLLM compared to those used in prior work. Results with models of similar scale show mixed performance, making it difficult to disentangle the improvement brought by the MLLM itself from the gains introduced by the proposed method. (Please provide analysis for model robustness on hyper-parameters and discussion on the effect of a stronger MLLM backbone.)\n\n\n**A2:** Thank you once again for your insightful question. To clarify the source of performance improvements, we conducted controlled experiments in Table 1 across different models (LLaVA-Video and Qwen2.5-VL) at both 7B and 72B scales. In both settings, incorporating ReAgent-V consistently leads to significant improvements over the corresponding base models. This indicates that the gains are not merely due to the use of a stronger backbone, but rather stem from our proposed reflection mechanism and reward-driven reasoning framework. These components enable the ReAgent-V framework to exhibit strong robustness across different model capacities."}, {"review_id": "aEBmcum36e", "reviewer": "Reviewer_KqtJ", "summary": "In the paper 'ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding' the authors tackle the problem of video understanding by proposing an agentic framework, based on reward generation via a reflection mechanism, that can also be used to filter high-quality frames for later fine-tuning, supervised or via reinforcement learning techniques.\nDifferently than current state-of-the-art methods, which use either high-quality annotations (expensive) or pretrained reward models (no real-time reasoning), the proposed algorithm aims to unlock iterative reasoning, self-correction and adaptability in more complex scenarios, not handled well by static approaches.\nThey test their framework on three tasks (visual understanding, visual reasoning and VLA alignment, 12 datasets in total) showing improved results over other baselines.", "questions": "How does frame selection reduce the inference time so much? (Table 3). Is the time required by this procedure offset by the reduction in computation?\n\nWhy is the amount of data used in Table 2 different for the various models? To validate the implicit claim (that ReAgent-V is better with less data) I'd like more information about how the other models behave with same amount of data, because it might be possible that the results would be similar with the same amount of data, or that ReAgent-V stays constant even after adding more data.", "limitations": "Not discussed in the main paper", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n(Quality) \n\t\tThe evaluation is extensive, using many different benchmarks. In general, the results are positive (but see weaknesses).\n\t\tThe evaluation is carried out on more than one tasks, including a control-based one.\n\n(Significance) \nI like the entropy-calibrated frame selection and I think it is an interesting approach with insights for the community.\n\n(Clarity)\n\t\tIn general the paper reads well and the method is explained well\n\n\nWeaknesses\n\n(Quality and significance) Although the evaluation is extensive, on many different benchmarks, I feel the results are somewhat of limited impact, especially in video understanding. This problem is worsened by not having any discussion on the statistical significance of the results (even though the authors claim, in the checklist, to provide one) and by limited insights provided (see later). Moreover, the experimental section is primarily focused on testing VLMs with and without ReAgent (for example, Table 1). This limits understanding of the significance of the method: how does this proposed framework improve over the stated limitations of other SOTA methods (static approaches, or approaches using RL etc.)? To experimentally evaluate this, more comparisons with methods employing such approaches should be provided (for example, taken from the related works, [13][29] etc ) instead of testing multiple VLMs.\n \n(Clarity)\nDiscussion/tests about the sensitivity of the entropy-based frame selection to the various hyperparameters is missing. There are some other problems: \n\ni) the section 2.2 on tool selection has little detail on how it actually works concretely. For example: 'the target agent proactively selects a subset of tools T ′ ⊆ T based on its reasoning needs'. How? \n\nii) Similarly, 2.3 could be improved. I understand the specific details are in the appendix, but I feel the main text should still include more details.  \n\niii)Small typo in line 200: missing space 'ReAgent-Vas...'\n\niv) In figure 5, I feel the question at the top has some errors: \"What is the telling when the burger ...\"  \n\n(Originality)\nBesides the entropy-based frame selection, the proposed algorithm provides limited insights. When proposing a modular architecture built on top other more fundamental blocks (like VLMs), I think it is important to explain in detail why the chosen 'blocks' were picked and compare this with the decisions made by other state-of-the-art methods, outlining in particular how the new ideas mitigate the weaknesses of other methods, especially those mentioned in the introduction/abstract. Why, for example, you use those three specific strategies in the reflection phase? Do other state-of-the-art methods do something similar? Were there weaknesses in other SOTA models that are mitigated by using these three particular strategies that cannot be tackled, for example, with an alternative revision strategy? The ablation of Figure 4 is not enough to answer this. Without comparisons or an in-depth discussion, as a researcher I get limited insights for possible future works, only the 'quantitative' results, and the choices made seem pretty arbitrary. I'd like to see more discussions in this sense.", "was_revised": true, "final_justification": "My main complaints were addressed and hence I believes it warrants a stronger score.", "rebuttal": "Thank you for your detailed and critical feedback - we have carefully reviewed each point and respectfully address them below:\n\n**Q1:** (Quality and significance) Although the evaluation is extensive, on many different benchmarks, I feel the results are somewhat of limited impact, … To experimentally evaluate this, more comparisons with methods employing such approaches should be provided (for example, taken from the related works, [1][2] etc ) instead of testing multiple VLMs.\n\nReferences:\n\n[1]  Videorag: Retrieval augmented generation over video corpus.\n\n[2]  Reflexion: Language agents with verbal reinforcement learning.\n\n**A1:** Thank you for the insightful question. Our evaluation focuses on whether large vision-language models (LVLMs) benefit from the ReAgent-V framework under different parameter settings. As shown in the results, ReAgent-V consistently outperforms its base models and surpasses most strong baselines, such as InternVL-2.5-8B and BIMBA-LLaVA.  We further conducted additional experiments comparing ReAgent-V with the retrieval-based VideoRAG approach and a Reflexion-inspired variant, as shown in the tables below. Although Reflexion [2] was originally designed for language tasks, we adapted its core mechanisms. All models were evaluated under identical settings for fairness. Due to space constraints, detailed implementation specifics will be included in the final version release.\n\nThe results show that ReAgent-V consistently outperforms [1], [2], and other baselines, demonstrating its effectiveness. All experiments were repeated three times, and our method achieves statistically significant improvements over models without ReAgent-V across all benchmarks (p < 0.01, two-tailed test), further validating its robustness.\n \n| Model (Backbone) | VideoMME (avg) | EgoSchema | LongBench |\n|------------------------|--------------|--------------|--------------|\n| ReAgent-V (Qwen2.5-VL-7B) | **60.7** | **61.9** | **54.3** |\n| VideoRAG (Qwen2.5-VL-7B) | 60.3 | 61.2 | 53.7 |\n| ReAgent-V (LLaVA-Video-7B) | **57.9** | **60.8** | **53.1** |\n| VideoRAG (LLaVA-Video-7B) | 57.2 | 60.3 | 52.6 | \n\n| Model (Backbone) | VideoMME (avg) | EgoSchema | LongBench |\n|------------------------|--------------|--------------|--------------|\n| ReAgent-V (Qwen2.5-VL-7B) | **60.7** | **61.9** | **54.3** |\n| Reflexion (Qwen2.5-VL-7B) | 50.6 | 53.2 | 43.8 |\n| ReAgent-V (LLaVA-Video-7B) | **57.9** | **60.8** | **53.1** |\n| Reflexion (LLaVA-Video-7B) | 46.7 | 50.5 | 42.8 | \n\n\nBeyond video understanding, we evaluate ReAgent-V on two additional tasks: Video LLM Reasoning and VLA Alignment. In Video LLM Reasoning, ReAgent-V outperforms the GRPO-trained model from [3] using only 45% of the training data. In VLA Alignment, our reflection-based reward replaces the hand-crafted cost function in GRAPE [4], yielding improved performance.\n\nReferences:\n\n[3] Video-r1: Reinforcing video reasoning in mllms, 2025.\n\n[4] GRAPE: Generalizing robot policy via preference alignment, 2025.\n\n---\n\n**Q2:** (Clarity) Discussion/tests about the sensitivity of the entropy-based frame selection to the various hyperparameters is missing.\n\n**A2:** Thank you for the question. We provide a robustness analysis of the two key hyperparameters in Eq(4) - the scaling factor $k$ and threshold $\\tau$ - using Qwen2-VL-7B on the short split of VideoMME.\n \nFirst, we fix $\\tau = 0.7$ and vary $k \\in \\{1, 2, 3, 4\\}$:\n\n**Table R1: Accuracy under different values of \\(k\\)**  \n| \\(k\\) | Accuracy (%) |\n|------|--------------|\n| 1    | 73.5         |\n| 2    | 71.5         |\n| 3    | 74.5         |\n| 4    | 72.8         |\n\nNext, we fix $k = 1$ and vary $\\tau \\in \\{0.5, 0.6, 0.7, 0.8\\}$:\n\n**Table R2: Accuracy under different values of $\\tau$**\n\n| $\\tau$ Value | Accuracy (%) |\n|---------|--------------|\n| 0.5     | 71.8         |\n| 0.6     | 72.3         |\n| 0.7     | 73.5         |\n| 0.8     | 71.2         |\n\nThe results show that our method is stable across a wide range of parameter values. Overly strict thresholds (e.g., $\\tau = 0.8$) may select too few frames and hurt performance. We adopt $k = 1$ and $\\tau = 0.7$ as default for a good balance of robustness and effectiveness, and will include this in the revised paper.\n\n---\n\n**Q3:** There are some other problems:\n\ni) the section 2.2 on tool selection has little … How?\n\nii) Similarly, 2.3 could be improved. I understand the specific details are in the appendix, but I feel the main text should still include more details.\n\niii)Small typo in line 200: missing space 'ReAgent-V as...'\n\niv) In figure 5, I feel the question at the top …burger ...\"\n\n**A3:**  Thank you for your careful reading and valuable comments. We will fix the typo on line 200 (“ReAgent-Vas...”) and revise unclear phrasing in Figure 5 (e.g., “What is the telling…”). We also acknowledge that Sections 2.2 and 2.3 lack implementation details. For tool selection, the base model determines which tools to invoke based on the question, the video content, and its own reasoning needs. In the revision, we will clarify how base models select tools based on the question and video content, and provide a more detailed explanation of tool selection and reflection strategies in the main text rather than in the appendix\n\n---\n\n**Q4:** (Originality) Besides the entropy-based frame selection, the proposed algorithm provides limited insights. … I get limited insights for possible future works, only the 'quantitative' results, and the choices made seem pretty arbitrary. I'd like to see more discussions in this sense.\n\n**A4:** Thank you again for the insightful comments. The three reflection strategies are designed to mitigate overconfidence in single-strategy responses, which often cause hallucinations and errors [1, 2]. Each strategy provides a distinct perspective, and their combination enables the model to integrate complementary signals, leading to more accurate answers. As shown in Figure 4, this combined approach outperforms individual strategies on benchmarks such as VideoMME, LongBench, and EgoSchema.\n\nTo further support this, we analyzed the VideoMME benchmark using ReAgent-V with Qwen2.5-VL-7B, focusing on the Temporal Perception task where reflection is frequently triggered. The average output confidences for the conservative, neutral, and aggressive strategies were 0.63, 0.71, and 0.45, respectively, with the aggressive strategy showing the highest variance. This suggests base models are less likely to produce conservative or aggressive responses due to lower confidence, but by explicitly prompting these strategies and merging their outputs, we can capture cases that single-strategy responses may overlook. ReAgent-V’s reward module not only generates reward reports to guide data selection for GRPO/DPO training but also directly improves base model reasoning through answer revision - a capability absent in prior agent frameworks. In the Video LLM Reasoning task, using ReAgent-V’s reward reports to filter GRPO training data yields more efficient performance gains than standard methods.\n\nReferences:\n\n [1] Analyzing and mitigating object hallucination in large vision-language models\n\n [2] VL-Uncertainty: Detecting hallucination in large vision-language model via uncertainty estimation\n\n---\n\n**Q5:** How does frame selection reduce the inference time so much? (Table 3). Is the time required by this procedure offset by the reduction in computation?\n\n**A5:** Thank you for the question. While ECRS-based frame selection introduces some overhead, it significantly reduces the number of frames passed to the vision-language model - the main inference bottleneck - resulting in efficiency gains that far outweigh the selection cost.\n\n---\n\n**Q6:** Why is the amount of data used in Table 2 different for the various models? To validate the implicit claim (that ReAgent-V is better with less data) … or that ReAgent-V stays constant even after adding more data.\n\n**A6:** Thank you for this interesting and careful question. In the Video LLM Reasoning application, ReAgent-V is used to filter GRPO training data via a reflection-based mechanism. As pointed out in [1], more challenging examples often lead to greater improvements in GRPO training. \n\nUsing the dataset from [2], we leveraged ReAgent-V’s reflection signals to identify difficult samples - those that triggered reflection - and retained only these difficult samples, comprising 45% of the original data. Training the base model from [2] on this subset under the same GRPO setup resulted in better performance than using the full dataset. This shows that ReAgent-V not only improves reasoning but also serves as an effective data filtering mechanism for reinforcement learning pipelines.\n\nTo further validate this, we conducted a control experiment by randomly selecting an equal-sized subset (52k samples) from the original training set. As shown below, the model trained on randomly selected data underperformed compared to both the full-data baseline and our reflection-based filtering:\n\n| Method               | Steps | #Data | VSI-Bench | VideoMMMU | MMVU | MVBench | TempCompass | VideoMME |\n|---------------------|--------|--------|------------|-------------|-------|-----------|----------------|-------------|\n| Vanilla GRPO [2]    | 16     | 116k   | 32.3       | 45.8        | 60.6  | 60.9      | 69.8           | 53.8        |\n| GRPO + Random select  | 16     | 52k    | 31.2       | 44.3        | 59.1  | 59.6      | 68.4           | 52.5        |\n| GRPO + ReAgent-V    | 16     | 52k    | **33.1**   | **47.9**    | **63.0** | **61.4**  | **70.3**       | **54.2**    |\n\nThese results highlight the importance of selecting informative training samples rather than relying on quantity, and confirm the utility of ReAgent-V as a high-quality data selector for downstream learning tasks.\n\n\nReferences:\n\n[1] Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement, 2025.\n\n[2] Video-r1: Reinforcing video reasoning in mllms, 2025."}, {"review_id": "sRguvrz1L3", "reviewer": "Reviewer_tdov", "summary": "The paper proposes a video understanding framework that iteratively alternates between a target agent generating the answer and a critic agent proposing feedback for refinement. The target agent is provided iterative feedback and adjusts its answer with a multi-perspective reflection strategy which aggregates altered versions of the original answer across varying intensities (conservative, neutral and aggressive), allowing the model to correct previous errors in reasoning. The paper also proposes an entropy-guided frame selection scheme, maximising the information present in frames that have high similarity to the original query.", "questions": "The paper is well written and verifies the effectiveness of its contributions through the ablations. However, some experimental validation and clarity is missing. To improve my score further, the authors should answer the following:\n\n1. Given the difference in model capacity and parameter count is unaccounted for, could the authors provide a fairer comparison with a similar capacity model, or further justification for the timing results presented in Table 3?\n2. Could the authors clarify whether the CLIP embedding extraction is performed online during inference or as an offline pre-processing step, as this impacts the interpretation of the reported timings?\n3. Could the authors provide an analysis of the failure modes that most frequently trigger the reflection mechanism? Specifically, which of the reasoning aspects outlined in the evaluation criteria tend to fail most often, necessitating refinement?\n4. Could the authors provide an analysis of the confidence distributions for each of the reflection types (conservative, neutral, and aggressive)? This would help clarify whether the prompting strategy is manipulating the agent's output confidence.", "limitations": "Yes - Added in the supplementary", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n- The paper supports its two main contributions: the reflection mechanism and an entropy-guided frame selection scheme, with the ablation studies showing improvement to the model's overall performance when both elements are added.\n- The core idea of employing iterative feedback between a target and critic agent is an interesting approach to addressing reasoning failures in video understanding. Furthermore, the proposed entropy-based is a strong idea for mitigating frame redundancy and increasing efficiency.\n- Beyond its primary application, the framework demonstrates broader utility by showing it can be used to identify challenging samples for fine-tuning. The framework enhances sample efficiency and boosts performance when fine-tuning for the video reasoning/general benchmarks.\n\n**Weaknesses:**\n- The validity of the timing comparisons presented in Table 3 is questionable. The proposed method is compared against baselines utilising GPT-4, but the analysis does not account for the differences in model capacity and parameter count. \n- The paper does not specify whether the extraction of CLIP embeddings is performed online during inference or offline as a pre-processing step. This also affects the reported timings in Table 3.\n- The paper introduces a multi-perspective reflection strategy but does not provide an analysis of the conditions under which this reflection is most often required. While the supplementary material indicates the evaluation criteria in the prompt, it remains unclear which aspects of reasoning most frequently fail, thereby triggering the refinement process. \n- An analysis of the confidence levels associated with each reflection type (conservative, neutral, aggressive) is notably absent. Without an examination of their respective confidence distributions, it is difficult to determine if the prompting strategy alters the output confidence. For instance, specifying an agent to be aggressive could cause over-confidence in its predictions and conversely under-confidence with the conservative agent.\n\n**Minor Weaknesses:**\n- Typo on Line 200: Should be a space between ReAgent-V and as\n- Line 272 appears to have an incorrect reference to Figure 4. Likely should be Figure 5.\n- The question \"What is the telling...\" in Figure 5 does not make sense and is inconsistent with the text on Line 287", "was_revised": true, "final_justification": "Thank you to the authors for their engagement during the discussions. Following this, the ambiguities were resolved and some valuable insights were offered to better understand the paper. Overall, I am happy to accept.", "rebuttal": "We are grateful for your comments and will address your concerns point by point in the following:\n\n**Q1:** Given the difference in model capacity and parameter count is unaccounted for, could the authors provide a fairer comparison with a similar capacity model, or further justification for the timing results presented in Table 3?\n\n**A1:** Thank you for the question. All inference time comparisons in Table 3 are conducted under the same model architecture and parameter scale. Taking LLaVA-Video-7B as an example, we control only whether the frame selection strategy is applied, in order to evaluate its impact on inference efficiency and accuracy. The results show that introducing the frame selection strategy leads to improvements in both efficiency and accuracy compared to the original model, demonstrating that the frame selection component in the ReAgent-V framework contributes positively to overall performance and ensures the objectivity and validity of the evaluation results.\n\n---\n\n**Q2:** Could the authors clarify whether the CLIP embedding extraction is performed online during inference or as an offline pre-processing step, as this impacts the interpretation of the reported timings?\n\n**A2:** Thank you for the question, and we apologize for not making this point clear in the original manuscript. All inference-time measurements reported in Table 3 include the CLIP feature extraction stage, which is performed online during inference rather than as an offline pre-processing step.\n\n---\n\n**Q3:** Could the authors provide an analysis of the failure modes that most frequently trigger the reflection mechanism? Specifically, which of the reasoning aspects outlined in the evaluation criteria tend to fail most often, necessitating refinement?\n\n**A3:**\nWe sincerely thank you for your thoughtful question. To better understand which reasoning dimensions most frequently trigger the reflection mechanism, we analyzed the reflection rates on the VideoMME benchmark using ReAgent-V with Qwen-2.5-VL 7B as the base model.\nOur analysis shows that when categorizing videos by length - Short, Medium, and Long - the corresponding reflection rates are 36%, 45%, and 67%, respectively. This suggests a positive correlation between reflection frequency and task difficulty, as current models tend to perform worse on longer videos.\nIn addition, we found that tasks related to Spatial Perception and Temporal Perception exhibit particularly high reflection rates, at 57% and 62% respectively. This indicates that current models are relatively weak in handling spatial and temporal reasoning in videos, often requiring multiple rounds of reflection to generate more accurate responses.\n\n---\n\n**Q4:** Could the authors provide an analysis of the confidence distributions for each of the reflection types (conservative, neutral, and aggressive)? This would help clarify whether the prompting strategy is manipulating the agent's output confidence.\n\n**A4:**  Thank you again for your insightful question. We provide additional quantitative evidence based on experiments using ReAgent-V with Qwen-2.5-VL 7B as the base model on the VideoMME benchmark. We focused on the task with the highest reflection rate - Temporal Perception - and analyzed the average output confidence under the three reflection strategies. The conservative strategy produced an average confidence of 0.63, the neutral strategy 0.71, and the aggressive strategy the lowest at 0.45, with larger variance.\n\nHowever, as shown in Figure 4, all three strategies are able to correct a portion of the answers after reflection. Moreover, combining all three strategies yields a higher overall accuracy than using any single strategy alone. This suggests that even though the conservative and aggressive strategies tend to generate lower confidence under standard model decoding, they may still introduce correct information in certain cases, ultimately leading to improved answer accuracy.\n\n---\n\n**Q5:**\nTypo on Line 200: Should be a space between ReAgent-V and as\nLine 272 appears to have an incorrect reference to Figure 4. Likely should be Figure 5.\nThe question \"What is the telling...\" in Figure 5 does not make sense and is inconsistent with the text on Line 287\n\n**A5:** \nReally thank you for your careful reading - we will correct the missing space on Line 200 (“ReAgent-V as”), fix the incorrect figure reference on Line 272 (should be Figure 5), and revise the unclear phrasing in Figure 5 to align with the question on Line 287 in the final version. We will also carefully review the manuscript to ensure overall consistency and clarity in the final version."}]

Author rebuttal:

Meta-review: This paper proposes to have a critic agent provides feedback to a target agent, which then refines its answer to video understanding problems using a multi-perspective reflection mechanism. The paper also introduces an entropy-calibrated frame selection method to improve performance and reduce computational cost.

All four reviewers found merit in the paper, appreciating the novelty of the agentic framework, the effectiveness of the proposed frame selection method, and the extensive experimental validation. Some of the initial concerns such as missing ablations and comparisons, clarity, and originality were largely addressed during the rebuttal phase, through additional experiments such as quantitative validation of ECRS, hyperparameter sensitivity analysis, and new comparisons against baselines. The paper overall makes valuable contributions to the venue.
Decision: Accept (poster)