Title: ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding
Venue: NeurIPS 2025
Accepted: yes
Keywords: Video understanding; Multi-agent framework; Reflective reasoning; VLA alignment; Video reasoning
Abstract: Video understanding is fundamental to tasks such as action recognition, video reasoning, and robotic control. Early video understanding methods based on large vision-language models (LVLMs) typically adopt a single-pass reasoning paradigm without dynamic feedback, limiting the model’s capacity to self-correct and adapt in complex scenarios. Recent efforts have attempted to address this limitation by incorporating reward models and reinforcement learning to enhance reasoning, or by employing tool-agent frameworks. However, these approaches face several challenges, including high annotation costs, reward signals that fail to capture real-time reasoning states, and low inference efficiency. To overcome these issues, we propose ReAgent-V, a novel agentic video understanding framework that integrates efficient frame selection with real-time reward generation during inference. These reward signals not only guide iterative answer refinement through a multi-perspective reflection mechanism—adjusting predictions from conservative, neutral, and aggressive viewpoints—but also enable automatic filtering of high-quality data for supervised fine-tuning (SFT), direct preference optimization (DPO), and group relative policy optimization (GRPO). ReAgent-V is lightweight, modular, and extensible, supporting flexible tool integration tailored to diverse tasks. Extensive experiments on 12 datasets across three core applications—video understanding, video reasoning enhancement, and vision-language-action model alignment—demonstrate significant gains in generalization and reasoning, with improvements of up to 6.9%, 2.1%, and 9.8%, respectively, highlighting the effectiveness and versatility of the proposed framework.

ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding

Abstract

Video understanding is fundamental to tasks such as action recognition, video reasoning, and robotic control. Early video understanding methods based on large vision-language models (LVLMs) typically adopt a single-pass reasoning paradigm without dynamic feedback, limiting the model's capacity to self-correct and adapt in complex scenarios. Recent efforts have attempted to address this limitation by incorporating reward models and reinforcement learning to enhance reasoning, or by employing tool-agent frameworks. However, these approaches face several challenges, including high annotation costs, reward signals that fail to capture real-time reasoning states, and low inference efficiency. To overcome these issues, we propose ReAgent-V, a novel agentic video understanding framework that integrates efficient frame selection with real-time reward generation during inference. These reward signals not only guide iterative answer refinement through a multi-perspective reflection mechanism-adjusting predictions from conservative, neutral, and aggressive viewpoints-but also enable automatic filtering of high-quality data for supervised fine-tuning (SFT), direct preference optimization (DPO), and group relative policy optimization (GRPO). ReAgent-V is lightweight, modular, and extensible, supporting flexible tool integration tailored to diverse tasks. Extensive experiments on 12 datasets across three core applications-video understanding, video reasoning enhancement, and vision-language-action model alignment-demonstrate significant gains in generalization and reasoning, with improvements of up to 6.9%, 2.1%, and 9.8%, respectively, highlighting the effectiveness and versatility of the proposed framework. Our code are available at https://github.com/aiming-lab/ReAgent-V .

1 Introduction

Video understanding, as one of the core tasks in computer vision, is widely applied in scenarios such as action recognition, video reasoning, and robotic control. Although significant progress has been made in recent years through the application of large vision-language models (LVLMs) and multimodal transformers, most existing methods adopt a static reasoning paradigm-that is, the model generates predictions directly from the input prompt and video in a single pass, without reflection or reward-based feedback to guide the reasoning process [10, 24, 4, 56, 6]. This design limits the model's adaptability in complex scenarios that demand iterative reasoning or task-specific feedback. For example, in tasks involving multi-step reasoning or reward-based evaluation, the model must reason step by step, verify intermediate results through reflection, and potentially use reward signals to assess the accuracy of each step. Without such mechanisms, static reasoning hampers the model's ability to self-correct and optimize its reasoning trajectory.

Entropy-Calibrated Frame Selection

Thesequenceofcolorchangeswhenthepersonpicksupthe

paper in thevideo?

OCR

Grounding DINO

ASR

Question

Scene GraphShareGPT4Video

Clip

Video

Frames

Prediction Flow

Application

Final

Answer

Answer

Inference

Score:...

Initial/Refined

1.Accuracy:

Target Agent

Answer

Critic Agent

Critical

2.Fine-grained:

3.

SFT

DPO

Questions

Evaluation

GRPO

Reflection

品

Report

Conservative

Neutral

Aggressive

Training

To improve video reasoning performance, previous research has mainly relied on two strategies: expanding high-quality annotated datasets for supervised fine-tuning (SFT) and introducing external reward models or reward templates to enhance the base model through reinforcement learning [10, 33, 54, 20]. While both approaches offer certain benefits, they also face notable limitations. On the one hand, obtaining high-quality annotations is often prohibitively expensive, limiting the scalability of methods that depend on increasing the volume of labeled data. On the other hand, using pretrained reward models for offline evaluation fails to capture the model's real-time reasoning state during inference, which can lead to suboptimal strategies or even reinforce flawed reasoning patterns. Although reward templates provide some form of real-time feedback, they are typically static and lack the adaptability needed to handle complex tasks or respond dynamically to changes during the reasoning process. For example, reward templates are typically predefined and are limited to providing feedback on multiple-choice answers or responses that follow specific templates, which makes it challenging to evaluate the reasoning process underlying a model's answer. More recently, approaches based on multi-model debate or tool-agent frameworks have been proposed to enhance reasoning capabilities [40, 9, 52]. However, these methods are often time-consuming and inefficient, with limited functionality, and they lack the ability to provide reward signals during inference.

To address these limitations, we propose an innovative agent-based video understanding framework, ReAgent-V, as shown in Figure 1. The core idea of our framework not only utilizes a dedicated frame selection module for efficient video understanding during inference, but also generates real-time reward signals. These signals help the model refine its answers, and at the same time, they can be used to filter high-quality data for SFT, DPO [33], and GRPO [34], enabling continuous optimization through learning during inference. To further enhance this process, ReAgent-V incorporates an intelligent reflection mechanism that prompts the target agent to revise its answers from conservative, neutral, and aggressive perspectives, where conservative only adjusts the final answer, neutral updates entities in the scene based on the input context, and aggressive modifies both the reasoning steps and the involved entities. This multi-perspective evaluation helps mitigate biases caused by model overconfidence and leads to more robust and diverse answer refinement. Additionally, ReAgent-V is designed with simplicity and extensibility in mind. It provides a modular architecture that allows users to customize the base models of both the target agent and the critic agent, as well as the task

templates. ReAgent-V supports a variety of applications, such as enhanced video understanding and video-based model rewarding. The former aims to improve the video reasoning capability of the existing base model, while the latter leverages reward feedback during the reasoning process to collect high-quality data, which is then used for training to further enhance the model's performance.

In conclusion, the key contribution of this work is ReAgent-V, the unified video understanding agentic framework capable of providing real-time rewarding during inference. Through extensive experiments on 12 datasets or tasks across three major applications-video understanding, video LLM reasoning, and vision-language action model alignment. ReAgent-V achieves performance improvements up to 6.9%, 2.1%, and 9.8%, respectively. Additionally, our study highlights the contribution of the frame selection mechanism in ReAgent-V and the intelligent reflection system, both of which are essential for improving the reasoning efficiency of the agent framework, reducing overconfidence bias, and enhancing the accuracy of reward feedback.

2 ReAgent-V

To improve video understanding and support dynamic reward feedback throughout the inference process, we propose a lightweight, general, and extensible agent framework, ReAgent-V. As illustrated in Figure 1, ReAgent-V enhances the reasoning process of the target agent through three key stages: entropy-calibrated frame selection, tool-augmented reasoning, and intelligent reflection with dynamically generated reward signals. First, the framework selects task-relevant video frames using an entropy-guided sampling strategy (§2.1), reducing redundancy and constructing a concise semantic context. Next, the target model performs initial reasoning by invoking appropriate external tools (§2.2). Finally, based on the reward signals provided by the critic agent, the target agent revises its initial answer from multiple perspectives-conservative, neutral, and aggressive-thereby improving the model's adaptability and robustness (§2.3). Detailed descriptions of each component are provided in the following sections, and the overall process is outlined in Algorithm 1.

2.1 Entropy-Calibrated Frame Selection

In this subsection, we introduce the keyframe selection strategy adopted in ReAgent-V. Common methods typically rely on generating captions for each frame or directly querying LVLMs such as GPT-4o to select keyframes that are most relevant to the prompt [49, 40]. However, such approaches are inefficient and slow in practice. Others focuses on CLIPbased selection [30, 48], where frames are chosen based on the CLIP score between each frame and the prompt. While efficient, this approach often fails on questions like 'Does

the person wash their hands before or after placing the cup?', where many frames contain a person, resulting in high CLIP scores even for irrelevant moments. This leads to the inclusion of redundant frames that are semantically unrelated to the question. We conducted a statistical analysis of this phenomenon, with detailed experiments reported in Appendix D. One example is shown in Figure 2, where the frames relevant to the question are manually annotated (as indicated by the red lines) after watching the video. We can observe that selecting frames solely based on the CLIP score tends to result in redundant frames.

To address this limitation, we propose a metric called the Entropy-Calibrated Relevance Score (ECRS), which jointly considers the semantic relevance of each frame to the query and the information entropy of each frame-i.e., the amount of information it contains-as illustrated by the orange line in Figure 2. Specifically, given a video V = { f 1 , f 2 , . . . , f N } consisting of N frames and a query Q , we first extract feature embeddings for each frame f i and the query Q using a shared CLIP encoder:

where e i and q represent the embedding vectors of the i -th frame and the query, respectively. The semantic similarity between each frame and the query is then computed by the cosine similarity s i = e ⊤ i q ∥ e i ∥∥ q ∥ , quantifies how well each frame f i aligns with the query Q . To incorporate frame-level

Algorithm 1 ReAgent-V Video Understanding Pipeline

Require: Video V ; Query Q ; Toolset T ; Prompt for critics agent Q c ; Reflection prompt for target agent { t c , t n , t a }

Ensure: Final answer A final; Evaluation report E

1: Stage 1: Frame Selection (§2.1)

2: Compute ECRS using Eqn. (3).

3: while there are frames meeting the threshold ECRS > k · α m · τ do :

4: Save the current round's selected frames F m .

5: Recompute ECRS in set F m .

6: Obtain the selected frame set F

7: Stage 2: Tool-Augmented Reasoning (§2.2)

8: Target agent select tools T ′ ⊆ T

9: Obtain tool information R via T i ( Q,F ) , where T i in T ′

10: Provide the target agent with F , Q , and R to obtain the initial answer A 0 .

11: Stage 3: Evaluation and Reflection (§2.3)

12: if Critic agent rejects A 0 then

13: Generate questions { q i } N i =1 .

14: Select new tools from T , update R update ← R , and generate evaluation report E .

15: Target agent generate reflected answers and confidence score ( A ( t ) , p ( t ) ) for t ∈ { t c , t n , t a } from R update , Q , F and E

16: if min( p ( c ) , p ( n ) , p ( a ) ) > 0 . 6 :

17: A final ← merged reliable parts of A ( t ) based on E and Q , where t ∈ { t c , t n , t a }

18: else : A final ← A ( t ) , where t = arg max t ∈{ t c ,t n ,t a } p ( t )

19: end if

20: return A final , E

information entropy, we denote the entropy of the i -th frame as H i . For color images, we compute the entropy separately for each RGB channel and take the average across channels:

where p ( i ) j,c represents the probability of pixel value j in channel c of the i -th frame. Finally, the ECRS for each frame is defined as:

where N is the total number of frames selected in the current frame set. This score jointly considers the similarity between each frame and Q , as well as the amount of information contained in each frame. After obtaining ECRS i , frame selection is performed iteratively. We set a base threshold τ and an iteration index m starting from 1, a frame f i is selected if:

The term k · α m serves as a scaling factor for the selection threshold. The scaling factor in each iteration allows the threshold to increase exponentially, thereby ensuring that frames selected in subsequent iterations have higher ECRS. In each iteration m , a set of candidate frames F m = { f 1 , f 2 , . . . , f c } is selected, where c denotes the number of frames chosen in the current round. For each frame in F m , we recalculate its score ECRS i using Eqn. (3), and apply Eqn. (4) again to select new frames. This process continues until no additional frames meet the threshold, in which case we return the frame set selected from the previous iteration. If the number of frames is below 32, we select the non-repetitive frames with the highest ECRS i from the previous round's frame set to complete the selection. As shown in the lower part of Figure 2, our method consistently increases the ECRS of key regions after iteration while suppressing those of irrelevant areas, leading to more accurate frame selection.

2.2 Tool-Augmented Reasoning

After selecting the key frames, to enhance the reasoning capability of ReAgent-V in video understanding tasks, we design a tool-augmented iterative reasoning mechanism. This mechanism allows

target agent to dynamically select and apply tools based on task requirements, iteratively refining its understanding through multiple rounds of information exchange, resulting in higher accuracy and interpretability. Specifically, given the user query Q , the selected key frame set F , and a tool set T , the target agent proactively selects a subset of tools T ′ ⊆ T based on its reasoning needs, and applies them to the user query Q and the selected key frame set F , for each tool T i in T ′ , generating intermediate tool results R i = T i ( Q,F ) . These intermediate results, along with Q and F , are fed into the target model, producing an initial answer A 0 .

2.3 Evaluation and Multi-Perspective Reflection

After the target agent generates an initial answer A 0 based on the selected frames F , query Q , and tool outputs R , the critic agent evaluates the quality of A 0 . If the answer is deemed unsatisfactory, the critic agent first generates a set of sub-questions { q 1 , q 2 , . . . , q l } based on the context to help localize potential errors. Using these sub-questions, the critic agent supplements the original toolset T ′ with additional tools as needed and applies them to F to obtain updated tool outputs R update. The critic agent then uses this updated information to produce a process reward for the initial answer in the form of an overall evaluation report E , which includes both a scalar reward score and structured feedback on initial answer A 0 (see Appendix C for examples of evaluation reports).

Based on the evaluation report, the target agent performs reflective reasoning from three strategic perspectives: conservative ( t c ), neutral ( t n ), and aggressive ( t a ). The conservative strategy focuses solely on adjusting the final answer, the neutral strategy updates the relevant entities in the scene (i.e., objects or elements related to the question) based on the given context, and the aggressive strategy revises both the reasoning steps and these entities. Prompt templates for each strategy are provided in Appendix B. Each strategy independently generates a revised answer A ( t ) and an associated confidence score p ( t ) , where t ∈ { t c , t n , t a } . The confidence score represents the model's estimated probability of the revised answer given the context:

Finally, if all confidence scores p ( t ) exceed a predefined threshold, the target agent aggregates the three revised answers { A ( t c ) , A ( t n ) , A ( t a ) } , extracts their common components, and removes inconsistencies to generate the final answer A final. Otherwise, the system selects the answer with the highest confidence.

3 Experiments

In this section, we demonstrate the effectiveness of ReAgent-V across three distinct video-related applications by addressing the following key questions: (1) Can ReAgent-V enhance performance in video understanding? (2) Does the feedback reward mechanism in ReAgent-V effectively select data that improves the reasoning capabilities of video LLMs? (3) Can the preference data generated through ReAgent-V improve alignment in preference optimization? (4) Are the frame selection and reflection mechanisms in ReAgent-V beneficial?

3.1 Applications and Experimental Setups

This section outlines three applications and describes the experimental settings, evaluation benchmarks, and baseline models. The applications include video understanding, video LLM reasoning, and vision-language-action (VLA) model alignment. Further details are provided below.

Video Understanding. This application evaluates the effectiveness of ReAgent-V in enhancing base models for video understanding. We apply ReAgent-V to a range of base models with varying sizes and architectures, including LLaVA-Video-7B/72B [56] and the Qwen series [46, 38] (Qwen2.5VL-7B/72B and Qwen2-VL-7B), and assess their performance across six widely used benchmarks: LongBench [39], NextQA [45], EgoSchema [31], LVBench [39], MLVU [59], and VideoMME [11]. To provide a comprehensive evaluation, we also include popular proprietary models [14, 37], opensource video-based LVLMs [4, 21, 55, 5, 41, 15], and general video agent frameworks [40]. For fair comparison, we re-evaluate all baselines (except proprietary models) using a similar number of video frames. During evaluation, both the target and critic models are set to the ReAgent-V-enhanced model itself. Detailed descriptions of the task prompt templates, the tool list, benchmark datasets,

and baseline models are provided in Appendix A and B.

Video LLM Reasoning. This application focuses on enhancing the reasoning capability of Video LLMs using ReAgent-V. Inspired by prior work [42] that demonstrates the effectiveness of using smaller, high-quality datasets to improve LLM or VLM reasoning, we leverage ReAgent-V as a data selection mechanism to identify valuable training samples. Specifically, we apply ReAgent-V to video samples from the Video-R1-260k [10] dataset, using the scores in E output by ReAgent-V during inference as the sample importance score. We retain videos and their original questions with importance scores lower than 5 (out of 10), as these cases tend to require more reflection during inference and are more challenging for the model. As such, we consider them more informative for training. The selected samples are then used to fine-tune the model using GRPO [34]. Following the experimental setup in [10, 9], we conduct comprehensive evaluations on 6 video reasoning datasets. Detailed descriptions of the benchmark datasets and baseline models are provided in Appendix A. VLAAlignment. This application investigates whether ReAgent-V can go beyond improving general video understanding to effectively benefit VLA alignment. Following [57], we aim to enhance the alignment of VLA models by using ReAgent-Vas the reward model, replacing the template-based reward function originally proposed in [57]. Specifically, we apply ReAgent-V to evaluate randomly collected rollouts from OpenVLA-7B in the SIMPLER environment [23]. These rollouts are assessed across multiple dimensions-success or failure, task completeness, stability, and accuracy-resulting in an overall evaluation score. These scores are then used to align the model's behavior via trajectorywise preference optimization (TPO) [57].

We evaluate the aligned models across four types of generalization: in-domain generalization, subject generalization, physical generalization, and semantic generalization, each consisting of multiple subtasks. Additionally, we benchmark performance against a diverse set of baselines trained under various paradigms and reward models, including OpenVLA-SFT, OpenVLA-DPO, and OpenVLATPO with GRAPE. All experiments are conducted under consistent settings, with each training paradigm using the same number of trajectory pairs to ensure fairness. Detailed descriptions of task setups, model configurations and hyperparameters are provided in Appendix A.

3.2 Main Results

We present the results of ReAgent-V across three applications: video understanding (Table 1), video LLM reasoning (Table 2), and VLA model alignment (Figure 3). Overall, ReAgent-V consistently yields substantial improvements across all tasks, achieving performance gains of 6.9%, 2.1%, and 9.8% in the respective applications. A detailed analysis for each application is provided below.

Video Understanding. In video understanding, ReAgent-V effectively enhances the performance of open-source LVLMs across various parameter sizes and architectures. For instance, without introduc-

ing additional frame inputs, ReAgent-V achieves average performance gains of 3.2% and 6.9% on multiple benchmarks compared to the original Qwen2.5-VL-72B and LLaVA-Video-72B, respectively. Notably, for LLaVA-Video-72B, ReAgent-V attains performance comparable to GPT-4o despite using significantly fewer input frames, highlighting its effectiveness and efficiency. Furthermore, at the same parameter scale, ReAgent-V surpasses other agentic methods such as VideoMemAgent [9] while using fewer frames and maintaining higher efficiency.

Video LLM Reasoning. In the video LLM reasoning, ReAgent-V outperforms vanilla GRPO by a relative average of 2.1%, while using only 45% of its training data. This demonstrates the effectiveness of data selection in enhancing video LLM reasoning, as well as the strength of ReAgent-V in guiding that selection. Moreover, ReAgent-V surpasses the SFT strategy, which was trained on the full 260k video-image dataset, achieving an average relative improvement of 4.3% across all reasoning and general benchmarks. These results collectively highlight the value of leveraging evaluation reports generated during ReAgent-V inference as rewards for guiding effective data selection.

VLA Alignment. In the VLA alignment task, ReAgent-V achieves a 9.8% overall improvement over the secondbest baseline, GRAPE (which uses a template reward), on SIMPLER under the same setting. Specifically, on the original task, ReAgent-V outperforms GRAPE by 9.0%. These results suggest that, compared to GRAPE-which relies on a predefined reward function to evaluate trajectory quality-ReAgent-V provides more accurate rewards for the trajectories generated by VLA models, thereby achieving better alignment.

3.3 Quantitative Analysis

In this section, we conduct comprehensive analysis to evaluate the effectiveness of incorporating entropy-calibrated frame selection and the multi-perspective reflection mechanism.

Analysis of Frame Selection Mech-

anism. In Table 3, we present the average per-sample inference time with and without frame selection, along with the corresponding performance differences across three benchmarks for various models. The results demonstrate that the frame selection strategy introduced in ReAgentV consistently yields substantial performance gains across LVLMs with diverse architectures and parameter scales, while simultaneously reducing inference time. This highlights the effectiveness of our entropy-calibrated

frame scoring strategy in selecting key frames that are both question-relevant and informative. Furthermore, we compare the performance of different video agent frameworks, and it is clear that

55.0

50.0

51.5

47.0

47.0

49.0

47.2

45.0

42.5

40.7 41.3

41.8

43.0

successrate

40.0

30.0

35.0

27.5

32.0

32.7

34.7

29.5

33.8

28.0

34.5

29.4

33.7

25.0

20.0

In-domain

Subject Generalization Physical Generalization Semantic Generalization

Average

OpenVLA-SFT

OpenVLA-DPO

OpenVLA-TPO GRAPE

OpenVLA-TPO ReAgent-V

put the eggplant

put the coke can on

put carrot (size:1.1)

put the green cube

into the basket

the towel

on the plate

ontotheyellowcube

VideoMME

LongBench

EgoSchema

Corr.Rate

Corr.Acc

60

(%)

60

Percentage

Percentage

40

40

20

20

tn

tc

ReAgent-V

t

ReAgent-V

What is the telling when the burger placed in the upper right video corner at the end of the video first appears?

A. Beef with spices came from Russia to Germany.

B. The steak began to be sandwiched between two pieces of bread.

C. Steak burgers spread throughout the United States.

D. The standardization of hamburgers."

Critic Questions

1.

What does the 1920 burger scene represent?

2.

Which restaurant chain appears behind the

upper-right burger?

3.

How is White Castle connected to the timeline?

To extract visible text like

"NINETEEN TWENTY"

to

anchor the timeline.

Initial answer : B

OCR

DET

Critic Agent

Target Agent

Target Agent

Add Tool:

To detect

burger

and

castle-like structure

as

concrete objects.

To interpret visual

symbols like

"1920" and the

castle as cues of

nationwide burger

spread.

Caption

Model

Structured Feedback:

The initial answer

B

reflects a

historical

interpretation

of the burger's origin.

However, the specific frame referenced …

by a drawing labeled

"Nineteen Twenty!"

and a

-

clearly situates the narrative at the point

of

commercial standardization

of

hamburgers in the United States.

...

Thus, while B …, it is not the correct

interpretation for the

visual and

temporal moment

prompted in the

question.

Scale Reward:

Visual Alignment: 2.0/5.0, reason: …

Temporal Accuracy: 1.5/5.0, reason: ...

Option Disambiguation: 2.0/5.0,

reason: …

Reasoning Specificity: 2.5/5.0, reason: …

Linguistic Precision: 4.0/5.0, reason: …

Total Score: 12.0/25.0

Evaluation report

Answer

: C

Confidence

: 0.9

Reasoning

: … steak

was placed

between bread…

Conservative

Neutral

Aggressive

Answer

: C

Confidence

: 0.9

Reasoning

: …

reflects the burger's

spread in the U.S. …

Answer

: D

Confidence

: 0.6

Reasoning

: The

1920 scene with

White Castle …

Final answer : C

Multi-Agent

Evaluation

Question

Reflection

ReAgent-V achieves significantly higher efficiency than other general video agent frameworks, further validating its effectiveness.

Analysis of Reflection Mechanism. To analyze the reflection mechanism in ReAgent-V, we conducted experiments across models with different architectures and parameter sizes. Specifically, we compare the performance of ReAgent-V with and without the reflection module on video understanding tasks. The version without reflection corresponds to the pipeline in Figure 4 where the agent directly performs tool-augmented reasoning based on selected frames. As shown in Table 4, incorporating the reflection module consistently improves the performance of various models on video understanding benchmarks. Furthermore, in Figure 4, we analyze the individual contributions of the three different perspectives - conservative, neutral, and aggressive - within the reflection process. Experimental results show that under the aggressive strategy, which allows modifying both the target agent's reasoning process and final answer, the corrected answers have the lowest accuracy. This may be because the model, in trying to follow the reflection template, mistakenly alters previously correct reasoning steps. In contrast, the conservative strategy, which only permits changing the final answer without modifying the reasoning, yields better correction performance, indicating that errors are more easily fixed when the original reasoning is preserved. By integrating three perspectives, ReAgent-V effectively mitigates errors caused by incorrect reasoning modifications while retaining the ability to correct mistakes, thereby enhancing the stability and overall accuracy of the reflection mechanism.

3.4 Case Study

To evaluate fine-grained video understanding, we present a case from the VideoMME dataset involving the question 'What is the video conveying when the burger first appears in the upper right corner?'-a scene marked by a burger, a '1920' timestamp, and a castle-like White Castle structure as is shown in Figure 5. Initially, using OCR and object detection, the model answered B ('The steak began to

be sandwiched between two pieces of bread'), missing key symbolic and temporal cues. Reflection via three sub-questions introduced a caption model, enabling semantic interpretation that revised the answer to C, recognizing the 1920 and castle as symbols of fast food's standardization. Multi-agent evaluation showed diverse views: the aggressive agent favored D (industrialization), while neutral and conservative agents supported C. Reward scores confirmed gains in reasoning specificity and visual alignment. This case demonstrates the complexity of symbol-rich, temporally grounded video QA and underscores the value of dynamic tool integration, reflection, and reward-driven evaluation.

4 Related Work

Multimodel LLMs for video understanding. Large vision-language models (LVLMs) have been widely applied in video understanding [14, 37, 4, 21, 55, 5, 25]. These models integrate tools like Optical Character Recognition (OCR) [32], Automatic Speech Recognition (ASR) [50], video object detection [29, 17], and video captioning [36, 4, 8, 47] to better interpret complex video content. Recent work has applied these tools to tasks like video question answering by fusing visual, textual, and auditory features [30, 16]. However, earlier approaches are static-producing answers in a single pass without iterative refinement-limiting their ability to handle multi-step reasoning and self-correction [30, 16]. To overcome this, newer frameworks employ retrieval-augmented generation to dynamically select relevant video segments [35], and multi-agent systems have emerged [40, 9], enabling modality-specialized agents to collaborate for real-time refinement and error correction, mimicking human-like iterative reasoning. Reflective reasoning further helps agents adapt outputs based on new information [53].

Nevertheless, most existing methods lack real-time reward feedback and struggle with refining predictions during inference. They also tend to overprocess video frames, lacking efficient frame selection [40, 9], and often rely heavily on the model's internal capacity or require retraining for effective reflection. In contrast, our framework, ReAgent-V, introduces a unified multi-agent system with real-time, inference-aware reward signals and a novel multi-perspective reflection mechanism. This enables dynamic refinement during inference and facilitates automatic selection of high-quality outputs for fine-tuning. Consequently, ReAgent-V achieves greater adaptability, interpretability, and performance across video understanding tasks-without the need for costly annotations or static reward templates.

Data-Centric VLM Reasoning Strategies. To enhance reasoning accuracy, methods like supervised fine-tuning (SFT) [7] and reinforcement learning [43] with human-defined reward models such as direct preference optimization (DPO) [33] and group relative policy optimization (GRPO) [34], have been developed. These approaches rely on fine-tuning base models with large annotated datasets and reward models that prioritize human-aligned outputs [7, 33, 34]. While promising, these strategies face critical challenges: annotating high-quality temporal data, such as action sequences or causal reasoning, is labor-intensive and costly, limiting scalability. Additionally, pre-trained reward models, typically used offline, cannot adapt to a model's real-time reasoning during inference [44, 28, 34, 18]. This becomes problematic in dynamic scenarios where feedback needs to be nuanced and responsive to the evolving reasoning process [51, 19, 1]. As a result, these reward models may fail to provide timely corrections or even reinforce erroneous reasoning patterns, hindering the model's ability to improve in real-world, dynamic tasks [60]. These limitations highlight the need for more adaptive and scalable approaches to enhance model performance in complex environments.

5 Conclusion

Wepresent ReAgent-V, a unified, agentic framework for video understanding that introduces inferencetime reward generation and multi-perspective reflective refinement. By combining lightweight entropy-calibrated frame selection with dynamic feedback, ReAgent-V moves beyond static, singlepass reasoning and enables effective tool-augmented self-correction. Experiments on 12 datasets across three applications-video understanding, video reasoning enhancement, and vision-languageaction model alignment-demonstrate consistent performance gains (up to 9.8%) with minimal overhead. We hope this framework inspires further exploration into efficient, self-improving agent systems capable of operating reliably in video understanding and reasoning.

References

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems , 35:23716-23736, 2022.

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 , 2025.

Yushi Bai, Xin Lv, Jiajie Zhang, Hongchang Lyu, Jiankai Tang, Zhidian Huang, Zhengxiao Du, Xiao Liu, Aohan Zeng, Lei Hou, et al. Longbench: A bilingual, multitask benchmark for long context understanding. arXiv preprint arXiv:2308.14508 , 2023.

Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. Sharegpt4video: Improving video understanding and generation with better captions. Advances in Neural Information Processing Systems , 37:1947219495, 2024.

Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271 , 2024.

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 24185-24198, 2024.

Pádraig Cunningham, Matthieu Cord, and Sarah Jane Delany. Supervised learning. In Machine learning techniques for multimedia: case studies on organization and retrieval , pages 21-49. Springer, 2008.

Songtao Ding, Shiru Qu, Yuling Xi, and Shaohua Wan. A long video caption generation algorithm for big video data retrieval. Future Generation Computer Systems , 93:583-595, 2019.

Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmented multimodal agent for video understanding. In European Conference on Computer Vision , pages 75-92. Springer, 2024.

Kaituo Feng, Kaixiong Gong, Bohao Li, Zonghao Guo, Yibing Wang, Tianshuo Peng, Benyou Wang, and Xiangyu Yue. Video-r1: Reinforcing video reasoning in mllms. arXiv preprint arXiv:2503.21776 , 2025.

Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. arXiv preprint arXiv:2405.21075 , 2024.

Randy W Heiland, M Pauline Baker, and Danesh K Tafti. Visbench: A framework for remote data visualization and analysis. In International Conference on Computational Science , pages 718-727. Springer, 2001.

Kairui Hu, Penghao Wu, Fanyi Pu, Wang Xiao, Yuanhan Zhang, Xiang Yue, Bo Li, and Ziwei Liu. Video-mmmu: Evaluating knowledge acquisition from multi-discipline professional videos. arXiv preprint arXiv:2501.13826 , 2025.

Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276 , 2024.

MdMohaiminul Islam, Tushar Nagarajan, Huiyu Wang, Gedas Bertasius, and Lorenzo Torresani. Bimba: Selective-scan compression for long-range video question answering. arXiv preprint arXiv:2503.09590 , 2025.

Soyeong Jeong, Kangsan Kim, Jinheon Baek, and Sung Ju Hwang. Videorag: Retrievalaugmented generation over video corpus. arXiv preprint arXiv:2501.05874 , 2025.

Licheng Jiao, Ruohan Zhang, Fang Liu, Shuyuan Yang, Biao Hou, Lingling Li, and Xu Tang. New generation deep learning for video object detection: A survey. IEEE Transactions on Neural Networks and Learning Systems , 33(8):3195-3215, 2021.

Zixuan Ke, Fangkai Jiao, Yifei Ming, Xuan-Phi Nguyen, Austin Xu, Do Xuan Long, Minzhi Li, Chengwei Qin, Peifeng Wang, Silvio Savarese, et al. A survey of frontiers in llm reasoning: Inference scaling, learning to reason, and agentic systems. arXiv preprint arXiv:2504.09037 , 2025.

Somnath Kumar, Yash Gadhia, Tanuja Ganu, and Akshay Nambi. Mmctagent: Multi-modal critical thinking agent framework for complex visual reasoning. arXiv preprint arXiv:2405.18358 , 2024.

Cheng Li, Jiexiong Liu, Yixuan Chen, and Yanqin Jia. Video-vot-r1: An efficient video inference model integrating image packing and aoe architecture. arXiv preprint arXiv:2503.15807 , 2025.

KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 , 2023.

Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22195-22206, 2024.

Xuanlin Li, Kyle Hsu, Jiayuan Gu, Oier Mees, Karl Pertsch, Homer Rich Walke, Chuyuan Fu, Ishikaa Lunawat, Isabel Sieh, Sean Kirmani, Sergey Levine, Jiajun Wu, Chelsea Finn, Hao Su, Quan Vuong, and Ted Xiao. Evaluating real-world robot manipulation policies in simulation. In 8th Annual Conference on Robot Learning , 2024.

Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122 , 2023.

Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. arXiv preprint arXiv:2408.15542 , 2024.

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision , pages 38-55. Springer, 2024.

Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos? arXiv preprint arXiv:2403.00476 , 2024.

Zijun Liu, Peiyi Wang, Runxin Xu, Shirong Ma, Chong Ruan, Peng Li, Yang Liu, and Yu Wu. Inference-time scaling for generalist reward modeling. arXiv preprint arXiv:2504.02495 , 2025.

Shengyu Lu, Beizhan Wang, Hongji Wang, Lihao Chen, Ma Linjian, and Xiaoyan Zhang. A realtime object detection algorithm for video. Computers & Electrical Engineering , 77:398-408, 2019.

Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: Visually-aligned retrieval-augmented long video comprehension. arXiv preprint arXiv:2411.13093 , 2024.

Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems , 36:46212-46244, 2023.

Ravina Mithe, Supriya Indalkar, and Nilam Divekar. Optical character recognition. International journal of recent technology and engineering (IJRTE) , 2(1):72-75, 2013.

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems , 36:53728-53741, 2023.

Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 , 2024.

Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems , 36:8634-8652, 2023.

Mingkang Tang, Zhanyu Wang, Zhenhua Liu, Fengyun Rao, Dian Li, and Xiu Li. Clip4caption: Clip for video caption. In Proceedings of the 29th ACM International Conference on Multimedia , pages 4858-4862, 2021.

Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530 , 2024.

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191 , 2024.

Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.08035 , 2024.

Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. In European Conference on Computer Vision , pages 58-76. Springer, 2024.

Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. Longllava: Scaling multi-modal llms to 1000 images efficiently via a hybrid architecture. arXiv preprint arXiv:2409.02889 , 2024.

Xiyao Wang, Zhengyuan Yang, Chao Feng, Hongjin Lu, Linjie Li, Chung-Ching Lin, Kevin Lin, Furong Huang, and Lijuan Wang. Sota with less: Mcts-guided sample selection for data-efficient visual reasoning self-improvement. arXiv preprint arXiv:2504.07934 , 2025.

Marco A Wiering and Martijn Van Otterlo. Reinforcement learning. Adaptation, learning, and optimization , 12(3):729, 2012.

Xiaobao Wu. Sailing ai by the stars: A survey of learning from rewards in post-training and test-time scaling of large language models. arXiv preprint arXiv:2505.02686 , 2025.

Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of questionanswering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 9777-9786, 2021.

An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115 , 2024.

Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont-Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10714-10726, 2023.

Zeyuan Yang, Delin Chen, Xueyang Yu, Maohao Shen, and Chuang Gan. Vca: Video curious agent for long video understanding. arXiv preprint arXiv:2412.10471 , 2024.

Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, et al. Re-thinking temporal search for long-form video understanding. arXiv preprint arXiv:2504.02259 , 2025.

Dong Yu and Lin Deng. Automatic speech recognition , volume 1. Springer, 2016.

Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances in neural information processing systems , 34:23634-23651, 2021.

Lu Zhang, Tiancheng Zhao, Heting Ying, Yibo Ma, and Kyusong Lee. Omagent: A multi-modal agent framework for complex video understanding with task divide-and-conquer. arXiv preprint arXiv:2406.16620 , 2024.

Wenqi Zhang, Ke Tang, Hai Wu, Mengna Wang, Yongliang Shen, Guiyang Hou, Zeqi Tan, Peng Li, Yueting Zhuang, and Weiming Lu. Agent-pro: Learning to evolve via policy-level reflection and optimization. arXiv preprint arXiv:2402.17574 , 2024.

Xingjian Zhang, Siwei Wen, Wenjun Wu, and Lei Huang. Tinyllava-video-r1: Towards smaller lmms for video reasoning. arXiv preprint arXiv:2504.09641 , 2025.

Y Zhang, B Li, H Liu, Y Lee, L Gui, D Fu, J Feng, Z Liu, and C Li. Llava-next: A strong zero-shot video understanding model. 2024.

Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXiv preprint arXiv:2410.02713 , 2024.

Zijian Zhang, Kaiyuan Zheng, Zhaorun Chen, Joel Jang, Yi Li, Siwei Han, Chaoqi Wang, Mingyu Ding, Dieter Fox, and Huaxiu Yao. GRAPE: Generalizing robot policy via preference alignment. In Workshop on Reasoning and Planning for Large Language Models , 2025.

Yilun Zhao, Lujing Xie, Haowei Zhang, Guo Gan, Yitao Long, Zhiyuan Hu, Tongyan Hu, Weiyuan Chen, Chuhan Li, Junyang Song, et al. Mmvu: Measuring expert-level multi-discipline video understanding. arXiv preprint arXiv:2501.12380 , 2025.

Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04264 , 2024.

Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 , 2019.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We accurately describe the scope and contributions of this paper in the abstract and introduction.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitaions of our work in appendix.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not contain any theoretical results or formal proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the experimental setup, training details, and hyperparameters in Appendix A to ensure reproducibility.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: All datasets are public datasets. We are working towards making our code available upon acceptance.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide detailed experimental setup, training details, and hyperparameters in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We discuss this in experiment section.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The computational resources used in the experiments are indicated in Appendix A.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper fully conforms with the code of ethics in every respect.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We discuss the broader impacts in appendix.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not release any models or datasets with high risk of misuse.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All original papers and codebases are appropriately cited in this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: LLMs are not used as an important, original, or non-standard component of the core method.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Experimental Setup

A.1 Dataset and Baselines

A.1.1 Video Understanding

The video understanding benchmarks evaluated in this work encompass six diverse and high-quality datasets, each meticulously curated to target a distinct and critical aspect of video-language comprehension. These benchmarks collectively span a wide range of tasks-including long-horizon reasoning, egocentric perspective analysis, multimodal alignment, and fine-grained visual-semantic understanding-thereby offering a comprehensive and rigorous evaluation of model capabilities across various real-world video scenarios.

LongBench [3] : Designed to evaluate long-horizon reasoning over extended video contexts, requiring models to maintain semantic coherence across many frames.

NextQA [45] : Focuses on temporal reasoning and causal understanding by presenting questions about action sequences and their logical consequences.

EgoSchema [31] : Adiagnostic benchmark for egocentric videos, assessing the ability to understand first-person perspectives and actions.

LVBench [39] : Tailored for multi-modal alignment, especially in complex settings where audio, text, and vision interplay is critical.

MLVU [59] : Emphasizes multi-task long video understanding, pushing models to perform a range of tasks such as captioning, question answering, and temporal ordering.

VideoMME [11] : A comprehensive benchmark that integrates various modalities-including subtitles and OCR-and is especially suited for evaluating fine-grained visual-semantic alignment and symbolic reasoning in videos.

To rigorously assess the performance of ReAgent-V, we compare it against a broad array of baseline models that span different capabilities and design philosophies. Proprietary closed-source models include GPT-4o [14] , known for its advanced multimodal reasoning, and Gemini-1.5-Pro [37] , optimized for long-context multimodal input. Among open-source video-language models, the evaluation includes:

ShareGPT4Video [4] : Enhances caption-driven video question answering by integrating ShareGPTstyle conversational supervision into multimodal video training. It leverages large-scale usergenerated captions and dialogues to align vision and language effectively, enabling better generalization to open-ended video queries.

VideoChat2 [21] : A chat-centric video-language model built to support multi-turn, conversational interactions grounded in video content. It emphasizes interactive understanding, with capabilities for dialogue continuity, temporal grounding, and multi-modal alignment, making it ideal for assistive agents and education scenarios.

LLaVA-Video [54] : An extension of the LLaVA framework adapted for video inputs, available in both 7B and 72B parameter versions. It employs frame-wise and temporal fusion techniques to convert video sequences into language-aligned embeddings, demonstrating strong performance on visual question answering and summarization tasks.

Qwen2 [2] and Qwen2.5-VL [2] families: Developed by Alibaba, these models exhibit high performance on both image and video understanding benchmarks through advanced multi-modal alignment. Qwen2.5-VL particularly excels in handling long-context and dense visual-textual reasoning tasks, supported by a unified visual-language architecture.

InternVL-2.5 [6] : A high-performing open-source model that scales both the architecture and training data to improve generalization. It incorporates vision-language alignment techniques, dense region grounding, and optimized pretraining routines to support both short and long video tasks with high efficiency.

BIMBA-LLaVA [15] : Introduces a selective scan compression mechanism tailored for long videos, where it prunes redundant frames while preserving key semantic content. This compression-aware training improves inference speed and memory usage while maintaining accuracy on temporal and contextual reasoning tasks.

Kangaroo [25] : Designed specifically for long-context video input, Kangaroo adopts hierarchical attention and memory-efficient tokenization strategies to process hundreds of frames. It is particularly effective for document-level video understanding, such as meeting summarization or sports analysis.

Long-LLaVA [41] : A variant of LLaVA optimized for efficient multi-frame processing. It integrates temporal coherence constraints and cross-frame attention mechanisms, making it capable of capturing nuanced motion patterns and temporal dependencies for improved video QA and description.

VideoAgent [40] :An agent framework for long-form video understanding that mimics human cognition through LLM-guided reasoning, CLIP-based retrieval, and VLM-driven state updates.

VideoMemAgent [9] : An agent framework for structured long-form video understanding that integrates unified temporal and object memory with tool-augmented reasoning, enabling multiround chain-of-thought inference across complex video content.

Across all these baselines, ReAgent-V almost outperforms in terms of accuracy, interpretability, and computational efficiency, largely due to its entropy-guided frame selection and multi-perspective reflection mechanism that support real-time reward-driven answer refinement.

Experiments are run on two H100-96GB GPUs with NVLink, using 64 frames per video. CLIP (ViT-L/14-336) is used for vision-language alignment. Videos are processed using decord and ffmpeg, with audio chunked for transcription. At inference, ReAgent-V performs tool-augmented reasoning by dynamically invoking OCR, ASR, DET and other modules. A multi-agent reflection strategy-comprising conservative, neutral, and aggressive evaluators-is followed by a meta-agent decision step.

A.1.2 Video LLM Reasoning

To investigate how ReAgent-V can enhance the reasoning capability of Video LLMs, we follow the experimental setup from [10] and apply ReAgent-V to filter the video portion of the Video-R1-260k dataset. Specifically, during inference, we retain samples with importance scores lower than 5 (out of 10) as these tend to be more challenging and thus more valuable for training. We then fine-tune the Qwen2.5-VL model on 8 NVIDIA H100 80GB GPUs using the filtered dataset and compare it against several baseline models defined in the paper. Our results show that the model achieves a 2.1% improvement in overall performance while using only 45% of the original training data, demonstrating the effectiveness of high-quality sample selection in enhancing video reasoning.

Benchmarks. We evaluate our model on six video benchmarks: VSI-Bench [12], VideoMMMU [13], MMVU[58], MVBench [22], TempCompass [27], and VideoMME [11]. The first three benchmarks focus primarily on video reasoning tasks that assess the model's ability to understand and reason over complex video semantics, while the latter three are general video understanding benchmarks involving a mix of perception and reasoning challenges. For MMVU, we evaluate using its multiple-choice question subset to ensure stability and consistency.

A.1.3 VLA Alignment

To construct a preference-based dataset for aligning vision-language-action (VLA) models, we collect and process rollouts from the SIMPLER environment [23]. Specifically, we construct a reward-ranked dataset for VLA alignment by sampling trajectories from four original in-domain tasks using the OpenVLA-7B-SFT-Simpler (OpenVLA-SFT) model, a supervised-finetuned baseline released by [57], with five rollouts per task. The remaining three generalization tasks are kept unseen and excluded from alignment. After data collection, we use ReAgent-V to evaluate each trajectory segment, assigning scalar reward scores based on task success, stability, completeness, and accuracy. For each task, we sample 20 best-vs-worst trajectory pairs, matched by task type and initial environment state. This yields 80 total trajectories, which are then partitioned into two RLDS-formatted datasets: a chosen set containing high-reward trajectories and a rejected set with low-reward counterparts.

We conduct all VLA alignment training on a single NVIDIA A100 GPU (80GB). The model is initialized from OpenVLA-SFT. Training employs LoRA (rank=32) with a learning rate of 2e-5 and dropout of 0.0. We use gradient accumulation with a step size of 4 to simulate larger batch sizes under

limited GPU memory. After training, the base model is merged with LoRA weights to obtain the final model. Our alignment process follows the Trajectory-wise Preference Optimization (TPO) [57] paradigm, where the objective is to learn a reward-aligned policy using the preference pairs described above. All results reported in the main paper are obtained after 6,800 steps of alignment training.

We evaluate all VLA models in the SIMPLER environment, covering four types of generalization: in-domain, subject generalization, physical generalization, and semantic generalization. In-domain consists of the four original tasks in the SIMPLER environment. Subject generalization includes three new tasks involving objects not seen during training. Physical generalization comprises eight tasks created by varying the width, height, and size of known objects. Semantic generalization includes four tasks where the original prompts are rephrased with synonymous instructions. Each taskset comprises several subtasks, and each subtask is evaluated over a fixed number of distinct episodes. The final metric is the average task success rate across episodes. For all baseline models-OpenVLA-SFT, OpenVLA-DPO, and OpenVLA-TPO with GRAPE-we use results reported by [57]. Our evaluation strictly follows the same experimental setup, adopting identical task definitions and environmental configurations to ensure a fair and consistent comparison.

B Evaluation Criteria and Prompts

B.1 Tool Selection Process

Tool Selection Prompt

[Task]

Carefully analyze the video content and identify exactly what information needs to be retrieved to support answering the given question. To answer the question step by step, list all the physical entities related to the question you want to retrieve, you can provide your retrieve request to assist you by the following JSON format:

[Tool Functions]

The Tool Factory supports the following types of retrieval via specialized tools:

Text Extraction: Tools: OCR , ASR -extract embedded text (e.g., signs, timestamps) and transcribe speech or narration.

Object Detection and Grounding: Tool: Grounding DINO -detects objects and aligns them with natural language prompts.

Multimodal Matching: Tools: CLIP -perform image-text alignment and retrieval based on semantic similarity.

Structured Scene Understanding: Tools: Scene Graph , Action Detector -construct structured graphs of object relations and detect visual actions.

Video Reasoning: Tools: ShareGPT4Video , VQA Model -enable video-based question answering and explanation generation.

Caption Generation: Tool: Captioning Model -generate scene-level textual descriptions.

Emotion and Identity Recognition: Tools: Face Recognition , Emotion Detector -detect character identities and their emotional states.

[Output format]:

Example 1:

Original question: 'How many blue balloons are over the long table in the middle of the room at the end of this video?' Options: 'A. 1. B. 2. C. 3. D. 4.' Output:

This section introduces the Tool Selection Prompt framework, designed to guide retrieval-aware video question answering through structured tool invocation. By decomposing a user question into

functional information needs-such as object detection, text extraction, or reasoning-the prompt enables precise specification of which tools (e.g., OCR, Grounding DINO [26], CLIP, Scene Graph, ASR) should be employed. The unified JSON output format supports downstream automation and tool chaining, ensuring modularity and extensibility. Multiple illustrative examples demonstrate how diverse question types can be mapped to appropriate retrieval functions and tools, from counting objects to recognizing emotional states or extracting narrated instructions. Notably, if additional visual tools or other modalities are required, users can simply modify the tool selection section to include the necessary tools, making this prompt a highly flexible and scalable interface for building robust multimodal QA pipelines.

B.2 Reflection Prompt

The Reflection Prompt framework is divided into three complementary strategies - Neutral B.2.1, Conservative B.2.2, and Aggressive B.2.3 - each offering a different level of intervention to revise initial video QA outputs. These prompts are designed to support multi-stage reasoning and grounded visual correction by reassessing object perception, answer validity, or full reasoning chains. Depending on the task format (e.g., open-ended QA, multiple choice), scoring mechanism (e.g., scalar reward, structured feedback), or desired reflection granularity, users can select the appropriate strategy: Neutral for perceptual correction, Conservative for stability-focused validation, and Aggressive for complete answer reconstruction. These templates can also be adapted or hybridized to support diverse evaluation workflows, enabling flexible deployment across multi-task QA pipelines.

B.2.1 Neutral Reflection Prompt

Entity-Centric Revision

[Task]

You are a neutral agent responsible for reassessing the initial model answer by correcting only visual misperceptions of scene elements. Your role is to revise the perceptual input (i.e., object/entity grounding) while preserving the original reasoning logic. Do not introduce any new reasoning steps.

[Reflection Requirement]

If the original answer is based on a misidentified visual entity, correct that grounding (e.g., object type, color, spatial position). Keep the interpretation process unchanged. This strategy focuses on refining what is seen , not how it is reasoned .

[Reflection Procedure]

Re-examine the video or image frames for object-level details (e.g., people, objects, colors, gestures).

Determine whether the initial answer failed due to incorrect or missing perception of visual entities.

Adjust the relevant scene elements accordingly (e.g., update object color, position, or identity).

Reuse the original reasoning chain, now applied to the corrected visual grounding.

[Evaluation Guidelines]

Remain neutral - do not assume the initial answer is incorrect unless there is clear evidence of perceptual error.

Do not modify the reasoning logic - preserve the original inference path.

Only revise the interpretation of visual content (i.e., what objects appear and their attributes).

[Input]

Question: {text}

Initial Answer: {initial_answer}

Eval Report: {structured feedback or scalar reward}

[Output]

Use the following format to provide the final revised answer after entity-level correction. Only output the revised answer and its confidence score - no explanations, no justifications, and no extra text of any kind.

Final Answer: ({free-form revised answer}, confidence score: 0-1)

B.2.2 Conservative Reflection Prompt

Answer-Focused Revision

[Task]

You are a conservative agent responsible for validating the initial model answer. Your role is to preserve the original answer unless there is irrefutable visual evidence that directly contradicts it. Do not revise or reinterpret the reasoning or scene elements unless the contradiction is absolute.

[Reflection Requirement]

Only revise the final answer itself-do not modify scene elements or reasoning logic. If the original answer remains potentially valid, it should be retained. This strategy aims for minimal intervention: conservative reflection focuses on maintaining output stability unless overwhelming visual contradiction is present.

[Reflection Procedure]

Re-examine the visual content for any direct contradiction to the initial answer.

Accept only literal, unambiguous visual cues that fully invalidate the original answer.

Do not alter object interpretation, scene structure, or logical flow.

Revise the final output only if the contradiction is undeniable and renders the original answer unsupportable.

[Evaluation Guidelines]

Retain the original answer if:

-Any uncertainty or ambiguity exists in the evidence

-Visual information lacks a clear, literal contradiction

-A reasonable observer could still accept the original answer

Revise the answer only if:

-The visual content presents overwhelming and explicit contradiction

-The revised answer exactly matches visible evidence without requiring interpretation

-The contradiction is strong enough to convince any neutral evaluator

[Input]

Question: {text}

Initial Answer: {initial_answer}

Eval Report: {structured feedback or scalar reward}

[Output Format]

Use the following format to provide the final decision after conservative validation. Only output the revised answer and your confidence score-no explanations, no justifications, and no extra text of any kind.

Final Answer:({free-form revised answer}, confidence score: 0-1)

B.2.3 Aggressive Reflection Prompt

Reasoning-Driven Revision

[Task]

You are an aggressive agent responsible for actively challenging the initial model answer. Your role is to revise both the reasoning process and the visual understanding in order to reconstruct a superior alternative.

[Reflection Requirement]

This strategy requires modifying both the reasoning steps and the associated scene entities. It involves the widest scope of change and is intended to completely overturn the original logic and rebuild a more accurate answer from scratch. Accept loose semantic alignment, reinterpret ambiguous scenes, and prioritize alternative perspectives over the original.

[Reflection Procedure]

Re-examine all video frames for cues that could support a different interpretation.

Modify the understanding of relevant visual entities (objects, attributes, spatial relations).

Reconstruct the reasoning chain based on the newly grounded observations.

Select a new answer that best fits the revised reasoning and evidence.

[Evaluation Guidelines]

Always replace the original answer - the default assumption is that it is suboptimal or incorrect.

Consider alternative answers even when based on partial, ambiguous, or abstract cues.

Allow semantic flexibility (e.g., 'canine' = 'dog', 'liquid' = 'water') and recontextualization.

Prioritize comprehensive reinterpretation to generate an improved final response.

[Input]

Question:

{text}

Initial Answer: {initial_answer}

Eval Report: {structured feedback or scalar reward}

[Output Format]

Use the following format to provide the final revised answer after full reasoning and entitylevel revision. Only output the revised answer and your confidence score - no explanations, no justifications, and no extra text of any kind.

Final Answer:

({free-form revised answer}, confidence score: 0-1)

B.2.4 Overall Reflection Prompt

Multi-Perspective Reflection Aggregation

[Task]

You are a specialized Meta-Agent for video question answering. Your role is to integrate the answers and confidence scores from three agents with different reflection strategies. Your goal is to synthesize a final answer by evaluating answer quality, confidence levels, and semantic overlap.

[Multi-Perspective Inputs]

Conservative Agent (Answer-Focused Reflection) Answer: {answer_conservative} , Confidence: {conf_conservative}

Neutral Agent (Entity-Centric Reflection) Answer: {answer_neutral} , Confidence: {conf_neutral}

Then:

Aggressive Agent (Reasoning-Driven Reflection) Answer: {answer_aggressive} , Confidence: {conf_aggressive}

[Decision Procedure]

Step 1 - High-Confidence Fusion

If all three confidence scores exceed their respective thresholds:

conf_conservative ≥ 0.6

conf_neutral ≥ 0.7

conf_aggressive ≥ 0.8

Combine the three answers {answer_aggressive}, {answer_aggressive}, {answer_aggressive}

Extract shared components and consistent semantic information

Remove contradictions or unsupported segments

Produce a final, coherent free-form answer that integrates the common insights

Step 2 - Confidence-Based Selection If one or more confidence scores fail to meet their thresholds:

Select the answer with the highest confidence score among the three agents

Use that agent's full revised answer as the final output

[Evaluation Criteria]

Semantic Overlap: Identify key phrases, facts, and themes that appear in multiple answers

Contradiction Removal: Discard any segments that directly conflict with others

Fluency: Ensure the final answer reads as a natural, well-formed sentence

[Input]

Question: {text}

Initial Answer: {initial_answer}

Agent Answers:

-{answer_conservative}

-{answer_neutral}

-{answer_aggressive}

Agent Confidences:

-{conf_conservative}

-{conf_neutral}

-{conf_aggressive}

[Output Format]

Use the following format to provide the final revised answer. Only output the revised answer -no explanations, no justifications, and no extra text of any kind.

Final Answer:

({free-form revised answer}

B.3 Critical Prompt

This section introduces two complementary evaluation prompts-Clarification Question Generation B.4 and Eval Report Generation B.5 - designed to support critical diagnosis of model answers in video question answering. The Clarification Prompt generates precise, targeted sub-questions when an answer is incomplete or ambiguous, uncovering reasoning or grounding gaps by referencing specific visual or contextual elements. It is particularly useful for both open-ended and multiple-choice QA tasks, helping localize errors without requiring immediate scoring. The Eval Report Prompt provides a structured scoring mechanism across five dimensions-visual alignment,

temporal accuracy, option disambiguation, reasoning specificity, and linguistic precision-yielding both qualitative feedback and a scalar reward useful for leaderboard tracking, model tuning, or reinforcement learning. Together, these prompts enable a flexible and task-adaptable evaluation framework: clarification questions are ideal for diagnosing failures and guiding revisions, while scalar scores support performance benchmarking. Depending on the evaluation goal-error localization, answer justification, or progress tracking-these prompts can be selectively applied or adapted, with clarification prioritized in open-domain QA and scoring emphasized in competitive or quantitative settings.

B.4 Critical Question Prompt

Clarification Questions Generation Prompt

[Task]

You are a critic agent tasked with evaluating the quality of the initial answer A 0 generated by the target agent, based on the provided question, context, and video content. If the answer is deemed unsatisfactory, your goal is to help localize potential errors by generating one or more sub-questions. These sub-questions mustshould be highly specific and firmly grounded in the visual or contextual evidence.

[Input Data]

Check if the answer fully addresses the question

Verify all key elements from context are included

Assess whether video content supports the answer

If not, raise sub-questions to expose missing or uncertain reasoning

[Clarification Guidelines]

If the answer is incomplete, generate 1-3 ultra-specific clarification questions following these rules:

Must start with: 'What', 'Where', 'When', 'Which', or 'How'

Must reference concrete elements from context/video

No vague pronouns ('it', 'they') - use specific nouns

Examples: 'Which timestamp shows the error?' or 'How many frames were processed?'

[Output Format]

answers)

B.5 Evaluation Report Prompt

Eval Report Generation

[Task]

You are a critic agent tasked with evaluating the quality of the initial answer A 0 generated by the target agent, using the given question and contextual information. Your goal is to provide structured diagnostic feedback by scoring the answer across multiple dimensions and computing a final scalar reward (0.0-1.0) based on the total score.

[Input Data]

Question:

{text}

Context: {context}

Initial Answer: {initial_answer}

[Evaluation Criteria]

Rate the answer on the following five dimensions (0.0-5.0 scale for each):

Visual Alignment : Is the answer aligned with visible video evidence?

Temporal Accuracy : Is the answer consistent with the timeline or timestamps?

Option Disambiguation : If multiple options are similar, does the answer clearly justify the selected one?

Reasoning Specificity : Is the reasoning clear, focused, and appropriately detailed?

Linguistic Precision : Is the answer grammatically correct and semantically accurate?

[Output Format]

Return a JSON object with detailed scoring and reasons for each dimension, plus the final total and normalized scalar reward:

}

B.6 Implementation Workflow of ReAgent-V

Figure 6 illustrates the complete inference workflow of ReAgent-V, a modular agent system designed for video-based question answering. The pipeline begins with unified initialization via load_default , followed by ECRS-based keyframe selection to reduce redundancy while preserving semantic relevance. A dictionary-based tool selection mechanism dynamically activates symbolic extraction tools (e.g., OCR, ASR, DET) based on the input query. Extracted textual context is merged into modal_strings and composed into a multimodal prompt for LLaVA-based initial answering. If critical gaps are identified, the system enters a reflective reasoning stage to revise the answer. Finally, an evaluation report is generated to assess the quality of both the initial and refined responses.

C More Visualization Results

C.1 Visualization of Frame Selection

These case studies qualitatively demonstrate the effectiveness of ECRS frame selection compared to uniform sampling across a range of video question answering scenarios. In each example, ECRS consistently captures frames that are more semantically aligned with the question, providing richer context and clearer evidence to support reasoning. For instance, in Figure 7, ECRS selects

Four FramesSampled Using Uniform Sampling

ONB

5

SAV

Questionaboutthevideo:Howdidthephotographerwhotookpicturesofthethree tourists in the video take the photo?

FourFramesSampled UsingUniformSampling

THE UNIVERSITY

OFQUEENSLAND

AUSTRALIA

CREATE CHANGE

Find out more at

uq.edu.au

Question about the video: What organ did the woman in the video

removefromthemedicalmodel?

frames that highlight the interaction between the photographer and tourists, directly addressing the question about how the photo was taken-unlike uniform sampling, which includes generic landscape shots. In Figure 8, ECRS captures the precise moment when the woman interacts with the medical model, effectively grounding the answer to the question about organ removal. Similarly, Figures 9 and 10 show that ECRS preserves critical moments involving text and actions-such as the student explaining his motivation for eating the banana or the banana-taping act that contextualizes the replaced painting-while uniform sampling misses or misaligns with these moments. Across all examples, ECRS provides temporally and semantically focused evidence that improves alignment between selected frames and the target question, validating its superiority in supporting visual reasoning. Note that although ECRS typically selects 20-40 informative frames per video for downstream processing, only four representative frames are visualized here by evenly sampling from the selected set, in order to maintain clarity and consistency in comparison.

C.2 Visualization of Evaluation Report

These four evaluation report cases demonstrate how ReAgent-V leverages visual tools and multi-agent collaboration to refine or correct initial answers by grounding them more accurately in visual and textual evidence. In Figure 11, the system correctly identifies the Dragon Boat Festival as the main theme of the video by combining OCR and DET outputs to interpret cultural symbols and scene elements. Figure 12 shows a quantitative reasoning adjustment, where the agent detects two birds and

Four Frames Sampled Using Uniform Sampling

A student ate a banana that

was part of an installation by the

DavidDatuna

Perfomanceartist

said he ate the fruit

because he was hungry.

beforereattaching

thebanana skin to the

wall using the same tape.

said he ate the fruit

because he was 'hungry.

but helateradded:damaging

a work of modem art could alsobeart:

FourFramesSampled UsingUniformSampling

Figure 10: ECRS captures the banana-taping act that replaced the shredded painting, providing better context.

revises the initial bird count from three to two, aligning the answer with grounded visual analysis. In Figure 13, the model originally misidentifies the cat's color, but after grounding the detected regions and reviewing visual attributes, the agents collaboratively revise the answer to the correct 'orange and white.' Finally, Figure 14 illustrates a geographic correction where the system initially mislabels the terrain as 'tropical,' but after integrating CLIP-based semantics and detailed grounding, it updates the answer to 'polar,' aligning better with snowy and icy visual cues. These examples highlight how reflective critique and grounded evidence improve factual accuracy and visual-textual alignment.

C.3 Visualization of VLA Alignment

Figure 15 visualizes six robotic manipulation tasks and highlights the progression from failure to success after applying reflection-guided reward correction during policy fine-tuning. In each task-carrot placement, spoon placement, eggplant basket placement, cube stacking, coke can placement, and sprite can placement-the left sequence illustrates the robot's initial failure to complete the action correctly, while the right sequence demonstrates the corrected behavior after fine-tuning. The visualizations show that the robot learns to adapt its positioning, trajectory, and precision based on reflective feedback. For example, in the carrot and spoon placement tasks, the

Question

Whichtraditional Chinesefestivalisprimarily showcased in thisvideo?

A. Dragon Boat Festival.

B.Spring Festival.

C.Mid-Autumn Festival.D.Double Ninth Festival

Toextractanyvisible

Todetectkeytraditional

Evaluation report

Chinese charactersthat

objects(e.g.,food, herbs,

mayexplicitlyreference

thefestivalname.

specificfestivals.

plants)associatedwith

The initial answerA.Dragon Boat Festival

StructuredFeedback:

accuratelyreflectsthevisualandtextual

evidencein thevideo.Earlyframes showzongzi,

dragonboatimagery,andcalamus...all

OCR

DET

strongly tied to this summerfestival.

OCRidentified characters like“端午"..offering

Target Agent

clear linguistic support.DETconfirmed symbolic

itemssuchaspaddlesand leaf-wrapped

foods...traditionallylinkedtotheoccasion.

Initial answer : A

While elements liketea sets orbannersmay

appear in otherfestivals,theconsistent

presenceof DragonBoat-specificvisuals

acrossscenesconfirmstheanswer.Thus,Ais

well-supportedbothsemanticallyand visually.

ScaleReward:

Visual Alignment:4.5/5.0,reason:...

TemporalAccuracy:4.0/5.0,reason:..

Critic Agent

ReasoningSpecificity:4.5/5.0,reason:...

Linguistic Precision:4.0/5.0, reason:...

Total Score: 21.0/25.0

Question

Howmanybirdsare there in thevideo??

A.1.B.2.

Answer: B

C.3.D.4.

Reasoning:..but

Confidence:0.7

one bird was

missed...

Conservative

Answer: C

Evaluation report

Reasoning:

Confidence:0.8

To locate specific object

instances matching the

The initial answer B. 2is a reasonable

Structured Feedback:

...partially

obscured....

"pq, ndoid

guessbasedonearly detections.Scene

Graph identified two bird-like objects,

and GroundingDINOconfirmedclearly

Neutral

Grounding

visible ones in the foreground...

Target Agent

DINO

However, a third bird—partially

occluded and blurredintheupper

Reasoning:distinct

Answer:C

Confidence:0.9

frame-was missed atfirst but later

bounding boxes

Initial answer :B

1. Are any birds partially hidden or flying in and

Critic Questions

represent'bird'

To detect and

confirmed throughrefinedgrounding and

objectrelation analysis (Object Detection

across frames...

and their spatial

and Relations).. The undercount likely

2.

Could closeproximity oroverlappinghave

relationships in

stemsfrom limited detectiongranularity

caused two birds to appear as one?

the scene

and overlapping instances.So while B.2

Aggressive

3.Do new birds appear as the video progresses?

is partially supported,itultimately

Multi-Agent

overlooks thefull scene content.

Evaluation

DET

Scale Reward:

VisualAlignment:3.5/5.0,reason:..

Temporal Accuracy:3.5/5.0,reason:.

Reflection

OptionDisambiguation:2.0/5.0,reason:

Final answer : C

Reasoning Specificity:2.5/5.0,reason:.

Critic Agent

Target Agent

Linguistic Precision:3.0/5.0,reason:..

Total Score:14.5/25.0

gripper initially misaligns with the target location but is corrected to center the object on the plate or towel. In the eggplant and cube stacking tasks, the robot improves its grasp and drop accuracy, successfully placing the object in or on the intended target. In the coke and sprite can tasks, the robot adjusts its vertical alignment and release timing to ensure stable placement. These results collectively demonstrate that reflection-driven fine-tuning enhances the robot's task completion reliability across diverse object manipulation scenarios.

Question

Whatcoloristhecatinthevideo?

A.Orange and white.

B.Black and white.

Answer: C

C.Orange.

D.Blue.

Confidence:0.5

Reasoning:...Given

the possibility of

"pess

Conservative

Answer: A

Confidence:0.8

Evaluationreport

Reasoning:

To match each answer

description like“An

To generate a scene-level

Structured Feedback:

The initial answer C. Orange is a

.. update the visual

facts I rely on ..

content of the video.

orange cat lying on a bed."

reasonable guess based on high-level

understanding. The Captioning Model

noted “an orange cat"...and CLIP

Neutral

CLIP

Caption

favored the“Orange”option.

Answer: A

Target Agent

Model

To detect and

localize a cat for

more detailed

However, the cat also has white

patches—on the underside and paws—

details missed by both tools...leading

Confidence:0.9

Reasoning:

information .

...missed bicolor

Initial answer : C

Critic Questions

1.Does the cat have any visible on

2.Is the cat's appearance actually

bi-colored,even if the dominant

the belly or legs?

descriptions

that may involve

multiple visual

traits

to a partial view of its appearance.

The tools offered broad but not detailed

nature. So while C. Orange is plausible,

it's incomplete given the visual evidence.

recognition...overlookingthebicolor

Aggressive

color is orange?

Grounding

Visual Alignment:3.5/5.0,reason:.

Scale Reward:

Multi-Agent

Evaluation

DINO

Temporal Accuracy: 3.5/5.0, reason: ...

Option Disambiguation:3.0/5.0,reason:

Reflection

Reasoning Specificity:3.5/5.0,reason:...

Linguistic Precision: 4.0/5.0, reason: ..

Final answer : A

Critic Agent

Target Agent

Total Score: 17.5/25.0

Question

Which geographical region's natural landscapes are primarily featured in this video?

A.Polar regions.

B.Tropical regions.

Answer:A

Confidence: 0.7

C.Subtropical regions.

D.Temperate regions.

Reasoning:

... revealed cold-

Conservative

Answer: A

Confidence:0.8

Evaluationreport

Reasoning:..more

To match each answer

content of the video.

choice with thevisual

description and terrain type.

To generate a scene-level

reflects a surface-level interpretation

The initial answer B. Tropical regions

Structured Feedback:

geographic

markers ...

based on blue waters and greenery seen

in early frames.. CLIP likely aligned

Neutral

CLIP

Caption

Model

these with tropical imagery.

However, later frames show icebergs,

Answer: A

Confidence: 0.7

Target Agent

Reasoning: the

To detect and

polar features missed by the initial

refined grounding of

Initial answer : B

CriticQuestions

align natural

tools...Theseelementsshift thescene

polar elements ...

2.Isthere evidence of wildlife,terrain,or

1.Do any frames suggest colder climates,

such as snow,glaciers,orice formations?

'icebergs'with

specificregions

elements like

whole.

reading,it‘s not correct for the video as a

clearlytowardpolarregions.

Thus, while B is a plausible partial

Aggressive

colortonestypicallyassociatedwiththe

otherregions?

Grounding

Visual Alignment: 2.5/5.0,reason:.

Scale Reward:

Multi-Agent

Evaluation

DINO

Temporal Accuracy: 2.0/5.0, reason:..

Option Disambiguation:1.5/5.0,reason:

Reflection

Reasoning Specificity: 2.5/5.0, reason:..

Linguistic Precision: 3.0/5.0,reason: .

Final answer : A

Total Score: 11.5/25.0

CriticAgent

TargetAgent

D Additional Experimental Results

D.1 Trends of Selection Scores Across Videos

Figure 16 illustrates the selection behavior of ECRS across various videos by comparing its score distribution with CLIP and Entropy-based baselines. The red bars indicate that ECRS consistently selects frames near local peaks, capturing semantically or visually informative moments. Compared to CLIP and Entropy score selections, ECRS demonstrates more temporally clustered choices that

(a) Carrot placement

(b) Spoon placement

(c) Eggplant basket placement

(d) Cube stacking

(e) Coke can placement

(f) Sprite can placement

Task: put the carrot on the plate (failed -> success)

Task: put the spoon on the towel (failed -> success)

Task: put the eggplant in the basket (failed -> success)

Task: stack the green cube on the yellow cube (failed -> success)

Task: put the coke can on the towel (failed -> success)

Task: put the sprite can on the towel (failed -> success)

align with high-compactness regions, indicating its ability to perform more focused and discriminative frame selection throughout the iterations.

D.2 Frame Index Trends Across Iterations

Figure 17 presents the iterative frame reduction behavior of ECRS across multiple videos, showing trends in average self-compactness (blue), GT-compactness (orange), and the number of selected frames (green). Self-compactness refers to the average index distance among the selected frame indices at each iteration, indicating how temporally clustered the selections are. GT-compactness measures the average index distance between selected frames and ground truth-relevant frames, reflecting how well the selected subset aligns with semantically important moments in the video. As iterations progress, both compactness metrics consistently decrease, demonstrating that ECRS not only reduces the number of selected frames but also improves their temporal tightness and alignment with ground truth. The number of selected frames drops sharply in early iterations and then stabilizes. This trend highlights ECRS's ability to efficiently eliminate redundant frames while preserving a compact, semantically meaningful subset aligned with human-annotated content.

Video ID:mAwgdX5VxGc

0.9

Question ID:624-1

estion ID: 622-2

S

0.8

EntropyScore

0.7

EntropyScore

0.6

0.5

0.4

0.3

ECRS (last iteration)

ECRS (last iteration)

Entropy Score

Normalize

0.4

0.3

0.5

0.2

ECRS(lrregularPeaks)

0.2

0

10

20

Frame Index

30

40

50

60

0

10

20

Frame Index

30

40

50

60

0.1

0

10

20

Frame Index

30

40

50

60

Score

Video ID: OncLRRLvdfk

Score

Video ID: pU_yyadYgG8

Video ID: oue5A-7Hpx4

S

0.9

0.8

0.7

Question ID: 634-1

Entropy Score

ECRS

0.8

0.9

0.6

Question ID:638-2

ECRS

Entropy Score

0.6

Question

2-1

Entropy Score

ECRS

0.5

0.4

0.4

0.3

0.3

0.3

10

20

30

40

50

60

0

10

20

30

40

50

60

10

20

30

40

50

60

Frame Index

Frame Index

Frame Index

Entropy Score

0.8

Entropy Score

0.8

Entropy Score

ECRS

ECRS

ECRS

0.4

0.4

0.4

0.2

0.0

0

10

20

Frame Index

30

40

50

60

0.0

10

20

Frame Index

30

40

50

60

0.0

0

10

20

FrameIndex

30

40

50

60

eoD:ID:637-3

1.0

1.0

VideolD: mvpbZGwZ6_4

QuestionID: 649-2

Entropy Score

Entropy Score

paz!

0.6

ECRS

ECRS

Entropy Score

0.0

10

20

Frame Index

30

40

50

60

20

Frame Index

OE

40

ECRS

0.0

10

20

Frame Index

30

40

50

60

0

1

2

3

4

5

6

7

Iteration

280

320

360

400

Average Distance

Video_ID: 0ay2Qy3wBe8

35

40

45

50

55

60

65

Number of Selected Frames

Number of Selected Frames

Average Distance (Self)

Average Distance (GT)

0

1

2

3

4

5

6

7

Iteration

240

280

320

360

Average Distance

Video_ID: PSt_op3fQck

45

50

55

60

Number of Selected Frames

Number of Selected Frames

Average Distance (Self)

Average Distance (GT)

0

1

2

3

4

Iteration

240

260

280

300

320

Average Distance

Video_ID: MPfuSfiZ96A

50

52

54

56

58

60

Number of Selected Frames

Number of Selected Frames

Average Distance (Self)

Average Distance (GT)

0

1

2

3

4

Iteration

330

340

350

360

Average Distance

Video_ID: NAahAX62MZ0

58

59

60

61

62

63

Number of Selected Frames

Number of Selected Frames

Average Distance (Self)

Average Distance (GT)

0

1

2

3

4

5

6

7

Iteration

280

320

360

400

Average Distance

Video_ID: 0ay2Qy3wBe8

35

40

45

50

55

60

65

Number of Selected Frames

Number of Selected Frames

Average Distance (Self)

Average Distance (GT)

0

1

2

3

4

5

6

7

Iteration

240

280

320

360

Average Distance

Video_ID: PSt_op3fQck

45

50

55

60

Number of Selected Frames

Number of Selected Frames

Average Distance (Self)

Average Distance (GT)

0

1

2

3

4

Iteration

240

260

280

300

320

Average Distance

Video_ID: MPfuSfiZ96A

50

52

54

56

58

60

Number of Selected Frames

Number of Selected Frames

Average Distance (Self)

Average Distance (GT)

0

1

2

3

4

Iteration

330

340

350

360

Average Distance

Video_ID: NAahAX62MZ0

58

59

60

61

62

63

Number of Selected Frames

Number of Selected Frames

Average Distance (Self)

Average Distance (GT)

D.3 Comparison of Frame Selection Methods Across Varying Input Frame Numbers

Figure 18 shows that ECRS consistently outperforms other frame selection strategies across all frame counts on the VideoMME dataset. As the number of input frames increases from 8 to 64, accuracy improves across all methods, and the performance gap between strategies remains consistent. While these trends are observed across different base models, the results shown here use LLaV A-Video-7B, which exhibits particularly strong performance, further highlighting the effectiveness of ECRS in enhancing model accuracy through more informative frame selection.

FrameSelectionComparisononVideoMMEDataset

CLIPScore

58

Entropy

ECRS (Ours)

Uniform

57

62

Accurac

56

55

54

8

16

32

64

NumberofFrames

D.4 Ablation of Visual Tools

Table 5 presents an ablation study on the impact of incorporating Visual Tools across different base models and benchmarks. Across all settings, enabling visual tools consistently improves performance on LongBench, EgoSchema, and VideoMME. The gains are particularly notable for LLaVA-Video-7B and Qwen2-VL-7B, where enabling visual tools leads to substantial improvements-for instance, +7.0 on LongBench and +4.5 on VideoMME for LLaVA-Video-7B. While the performance gap is smaller for larger models like LLaVA-Video-72B and Qwen2.5-VL-72B, improvements are still evident on LongBench and VideoMME. These results indicate that visual tools provide valuable auxiliary signals that enhance multimodal reasoning, especially for smaller or less capable base models.

Limitation

Despite employing keyframe selection, handling long videos remains challenging due to information redundancy and complex semantic relationships, which hinder the model's ability to achieve comprehensive and accurate understanding and reasoning. Additionally, the reflection and evaluation mechanisms rely on heuristic-based rules or templates, lacking adaptive and end-to-end learning capabilities, which may limit the effectiveness of automatic error correction.

Social impact

Our method enhances video understanding and reasoning, which can benefit education, assistive technologies, and human-computer interaction. However, potential risks such as privacy concerns and the propagation of dataset biases should be carefully considered during deployment.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a video understanding framework that iteratively alternates between a target agent generating the answer and a critic agent proposing feedback for refinement. The target agent is provided iterative feedback and adjusts its answer with a multi-perspective reflection strategy which aggregates altered versions of the original answer across varying intensities (conservative, neutral and aggressive), allowing the model to correct previous errors in reasoning. The paper also proposes an entropy-guided frame selection scheme, maximising the information present in frames that have high similarity to the original query.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The paper supports its two main contributions: the reflection mechanism and an entropy-guided frame selection scheme, with the ablation studies showing improvement to the model's overall performance when both elements are added.\n- The core idea of employing iterative feedback between a target and critic agent is an interesting approach to addressing reasoning failures in video understanding. Furthermore, the proposed entropy-based is a strong idea for mitigating frame redundancy and increasing efficiency.\n- Beyond its primary application, the framework demonstrates broader utility by showing it can be used to identify challenging samples for fine-tuning. The framework enhances sample efficiency and boosts performance when fine-tuning for the video reasoning/general benchmarks.\n\n**Weaknesses:**\n- The validity of the timing comparisons presented in Table 3 is questionable. The proposed method is compared against baselines utilising GPT-4, but the analysis does not account for the differences in model capacity and parameter count. \n- The paper does not specify whether the extraction of CLIP embeddings is performed online during inference or offline as a pre-processing step. This also affects the reported timings in Table 3.\n- The paper introduces a multi-perspective reflection strategy but does not provide an analysis of the conditions under which this reflection is most often required. While the supplementary material indicates the evaluation criteria in the prompt, it remains unclear which aspects of reasoning most frequently fail, thereby triggering the refinement process. \n- An analysis of the confidence levels associated with each reflection type (conservative, neutral, aggressive) is notably absent. Without an examination of their respective confidence distributions, it is difficult to determine if the prompting strategy alters the output confidence. For instance, specifying an agent to be aggressive could cause over-confidence in its predictions and conversely under-confidence with the conservative agent.\n\n**Minor Weaknesses:**\n- Typo on Line 200: Should be a space between ReAgent-V and as\n- Line 272 appears to have an incorrect reference to Figure 4. Likely should be Figure 5.\n- The question \"What is the telling...\" in Figure 5 does not make sense and is inconsistent with the text on Line 287\n\nQUESTIONS:\nThe paper is well written and verifies the effectiveness of its contributions through the ablations. However, some experimental validation and clarity is missing. To improve my score further, the authors should answer the following:\n\n1. Given the difference in model capacity and parameter count is unaccounted for, could the authors provide a fairer comparison with a similar capacity model, or further justification for the timing results presented in Table 3?\n2. Could the authors clarify whether the CLIP embedding extraction is performed online during inference or as an offline pre-processing step, as this impacts the interpretation of the reported timings?\n3. Could the authors provide an analysis of the failure modes that most frequently trigger the reflection mechanism? Specifically, which of the reasoning aspects outlined in the evaluation criteria tend to fail most often, necessitating refinement?\n4. Could the authors provide an analysis of the confidence distributions for each of the reflection types (conservative, neutral, and aggressive)? This would help clarify whether the prompting strategy is manipulating the agent's output confidence.\n\nLIMITATIONS:\nYes - Added in the supplementary", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn the paper 'ReAgent-V: A Reward-Driven Multi-Agent Framework for Video Understanding' the authors tackle the problem of video understanding by proposing an agentic framework, based on reward generation via a reflection mechanism, that can also be used to filter high-quality frames for later fine-tuning, supervised or via reinforcement learning techniques.\nDifferently than current state-of-the-art methods, which use either high-quality annotations (expensive) or pretrained reward models (no real-time reasoning), the proposed algorithm aims to unlock iterative reasoning, self-correction and adaptability in more complex scenarios, not handled well by static approaches.\nThey test their framework on three tasks (visual understanding, visual reasoning and VLA alignment, 12 datasets in total) showing improved results over other baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n(Quality) \n\t\tThe evaluation is extensive, using many different benchmarks. In general, the results are positive (but see weaknesses).\n\t\tThe evaluation is carried out on more than one tasks, including a control-based one.\n\n(Significance) \nI like the entropy-calibrated frame selection and I think it is an interesting approach with insights for the community.\n\n(Clarity)\n\t\tIn general the paper reads well and the method is explained well\n\n\nWeaknesses\n\n(Quality and significance) Although the evaluation is extensive, on many different benchmarks, I feel the results are somewhat of limited impact, especially in video understanding. This problem is worsened by not having any discussion on the statistical significance of the results (even though the authors claim, in the checklist, to provide one) and by limited insights provided (see later). Moreover, the experimental section is primarily focused on testing VLMs with and without ReAgent (for example, Table 1). This limits understanding of the significance of the method: how does this proposed framework improve over the stated limitations of other SOTA methods (static approaches, or approaches using RL etc.)? To experimentally evaluate this, more comparisons with methods employing such approaches should be provided (for example, taken from the related works, [13][29] etc ) instead of testing multiple VLMs.\n \n(Clarity)\nDiscussion/tests about the sensitivity of the entropy-based frame selection to the various hyperparameters is missing. There are some other problems: \n\ni) the section 2.2 on tool selection has little detail on how it actually works concretely. For example: 'the target agent proactively selects a subset of tools T ′ ⊆ T based on its reasoning needs'. How? \n\nii) Similarly, 2.3 could be improved. I understand the specific details are in the appendix, but I feel the main text should still include more details.  \n\niii)Small typo in line 200: missing space 'ReAgent-Vas...'\n\niv) In figure 5, I feel the question at the top has some errors: \"What is the telling when the burger ...\"  \n\n(Originality)\nBesides the entropy-based frame selection, the proposed algorithm provides limited insights. When proposing a modular architecture built on top other more fundamental blocks (like VLMs), I think it is important to explain in detail why the chosen 'blocks' were picked and compare this with the decisions made by other state-of-the-art methods, outlining in particular how the new ideas mitigate the weaknesses of other methods, especially those mentioned in the introduction/abstract. Why, for example, you use those three specific strategies in the reflection phase? Do other state-of-the-art methods do something similar? Were there weaknesses in other SOTA models that are mitigated by using these three particular strategies that cannot be tackled, for example, with an alternative revision strategy? The ablation of Figure 4 is not enough to answer this. Without comparisons or an in-depth discussion, as a researcher I get limited insights for possible future works, only the 'quantitative' results, and the choices made seem pretty arbitrary. I'd like to see more discussions in this sense.\n\nQUESTIONS:\nHow does frame selection reduce the inference time so much? (Table 3). Is the time required by this procedure offset by the reduction in computation?\n\nWhy is the amount of data used in Table 2 different for the various models? To validate the implicit claim (that ReAgent-V is better with less data) I'd like more information about how the other models behave with same amount of data, because it might be possible that the results would be similar with the same amount of data, or that ReAgent-V stays constant even after adding more data.\n\nLIMITATIONS:\nNot discussed in the main paper", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper proposes an agentic video understanding framework. It involves a frame selection to pick salient input frames. It also provides a tool factory to assist model for video understanding. To supervise the model, it designs a critic agent for evaluating model results and providing evaluation report for improvement guidance. The target agent then refine based on three strategies to produce better predictions.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper proposes a new framework for agentic video understanding, and shows the framework is versatile and can support both test-time improvement and training time data construction.
- The proposed entropy-based frame selection is interesting and has demonstrated consistent improvement.
- The paper writing is easy to follow with clear structure.
- The experimental results are reported for three settings to verify method effectiveness.

Weakness:
- The selection threshold used in Eq(4) involves multiple parameters. It is unclear if the method is robust to those parameters or requires careful parameter tuning.
- In Table 1, the best performance is achieved with Qwen 2.5 72B, which is a larger MLLM than those used in [8]. Results with similar scale of MLLM shows mixed performance. It makes it hard to understand the improvement from the MLLM vs improvement from the method.

QUESTIONS:
Please provide analysis for model robustness on hyper-parameters and discussion on the effect of stronger MLLM backbone.

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ReAgent-V, a reward-driven multi-agent framework for video understanding. The framework consists of three parts. First, it selects frames relevant to the query by an entropy-calibrated selection strategy. Second, it invokes various tools (e.g., text extraction, object detection) to assist the reasoning process. Finally, it introduces a multi-perspective reflection mechanism, where a critic agent produce suggestions to guide the target agent to refine the answer. The paper compare ReAgent-V to multiple video understanding baselines on multiple video tasks. The paper also quantitatively and qualitatively analyzes the contributions of each component\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1.The proposed framework addresses the important limitation of static, single-pass video reasoning in large vision-language models. By integrating real-time reward generation and multi-perspective reflection, ReAgent-V enables dynamic, self-correcting, and tool-augmented inference, which is highly relevant for complex video understanding scenarios.\n2.The experiments are solid and comprehensive. The authors conduct extensive experiments on 12 datasets across three applications—video understanding, video LLM reasoning, and vision-language-action alignment. Results show clear performance gains over prior baselines. The paper also includes ablations for frame selection and reflection mechanisms, showing meaningful improvements in both accuracy and efficiency.\n3.The agent frame work is extensible to different backbone models.\n\nWeaknesses\n1.The paper lacks quantitative analysis on the accuracy of ECRS. There’s no data showing how well frames selected by ECRS align with actual keyframes related to the query.\n2.The paper uses prompts to control the behavior of the three different reflection agents (conservative, neutral, and aggressive), but it doesn’t provide quantitative evidence showing whether these agents truly exhibit distinct behavior patterns.\n\nQUESTIONS:\n1.Could the authors provide more quantitative results on the selection quality of ECRS? Are there failure modes where ECRS excludes or over-selects critical frames? \n\n2.Could the authors clarify how the conservative, neutral, and aggressive reflection strategies differ in practice by cases? How robust are these strategies to prompt during reflection?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-05"}