MR. Video: MapReduce as an Effective Principle for Long Video Understanding
Authors: Ziqi Pang
Yu-Xiong Wang
Venue: NeurIPS 2025 poster ()
Keywords: LLM agents
long video understanding
Primary area: applications

Abstract: The fundamental challenge of long video understanding, e.g., question answering, lies in the extensive number of frames, making it infeasible to densely understand the local details while comprehensively digest the global contexts, especially within a limited context length. To address this problem, our insight is to process short video segments individually and combine these segment-level analyses into a final response. This intuition is noted in the well-established MapReduce principle in big data processing and is naturally compatible with inference scaling at the system level. Motivated by this, we propose MR. Video (pronounced as "mister video"), a long video understanding framework adopting the MapReduce principle. We define the standard operations of MapReduce in a long video understanding context: the Map steps conduct independent and sequence-parallel dense perception on short video segments, covering local details, while the Reduce steps comprehensively aggregate the segment-level results into an answer with global contexts. Thanks to the low cost and convenience of building video agents, we instantiate such Map and Reduce operations as an effective video agent capable of attending to local details and global contexts. Based on such abilities, we further introduce two critical yet previously under-explored long video understanding designs: (a) consistent character/object names in the captions, benefiting the reasoning of actions and stories across long horizons; (b) question intention analysis, which changes the key-frame retrieval in previous video agents to localizing the relevant information via jointly reasoning the whole video contexts and questions. Our MR. Video achieves a >7% accuracy improvement on the challenging LVBench over state-of-the-art video agents and vision-language models (VLMs) and demonstrates a clear advantage on multiple long video benchmarks, highlighting the potential of the MapReduce principle. The code is at https://github.com/ziqipang/MR-Video}{https://github.com/ziqipang/MR-Video.

MR. Video: MapReduce as an Effective Principle for Long Video Understanding
Ziqi Pang Yu-Xiong Wang
University of Illinois Urbana-Chama Champaign {ziqip2, yxs]@illinois
Abstract
The fundamental challenge of long video understanding, e.g., question answering, lies in the extensive number of frames, making infeasible to densely understand the local details while comprehensively digest the global contexts, especially within a limited context length. To address this problem, our insight is to process short video segments individually and combine these segment-level analyses into final response. This intuition is noted in the well-establiseed MapReduce principle big data processing and is naturally compatible with inference scaling the system level. Motivated by this, we propose MR. Video', long video understanding framework adopting the MapReduce principle. We define the standard operations MapReduce in a long video understanding context: the Map steps conduct independent and sequence-paralle dense perception on short video segments, covering local details, while the Reduce steps comprehensively aggregate the segment- -evell results into answer with global contexts. Thanks to the low cost and convenience of building video agents, we instantiate such Map and Reduce operations as effective video agent capable of attending to local details and global contexts. Based on such abilities, we further introduce two critical yet previously under-explored long video understanding designs: (a) consistent character/object names in the captions, benefiting the reasoning of actions and stories across long horizons; (b) question intention analysis, which changes the key-frame retrieval previous video agents to localizing the relevant information via jointly reasoning the whole video contexts and questions. Our MR. Video achieves > accuracy improvement on the challenging LVBench over state-of-the video agents and Langua-anguage models (VLMs) and demonstrates clear advantage on multiple long video benchmarks, highlighting the potential of the MapReduce principle. The code is at http://ggtuuub.oo/iiiiaagg////// Video.
Introduction
Considering challenging example for long video understanding (Fig. left): suppose are watching a fast-paced sports video and wanting to count the number of specific events, e.g., goals by player, model should carefully go through every action to inspect the criteria of "a goal by No. 11," and then comprehensively aggregate across the whole video duration, especially when the number of events is as large as 200. Such an example reveals the fundamental challenge in long video understanding: how to digest global contexts while perceiving local details.
Unfortunately, existing 61] that rely on using large language models (LLMs) to process video tokens are limited in context lengths. So they are forced to sample frames sparsely or compress tokens (Fig. ((a)), losing the dense local details, e.g., missing the events in the example video or failing to recognize the correct person. Although video agents [8, 42, 45, 53] emerge to bypass the VLMs' context length limitations via strategically selecting small set of video clips to perceive, they sacrifice the other aspects of long video understanding: (1) they generally rely on multi-round exploration of video segment selection
se-euueee vision-language models (VLMs) [16, 17, 24, 26, 27, 28,
pronounced as "mister video"
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Seq-to-Seq VLMs
(b) Video Agents
<img>
Fill Conbest Langths... <imgg <img>
15:40
50
Multi-modal
Frame Selection
Analysts
Frame Selection
Analysts
(c) MapReduce (Ours)
(d)
Scalabke & Seq-Parallel
Contest-Lenn Unlimiting
Global Info
VLMs
X
v
This video features how many No.
Map
Nap
Incependenn ndlent
Map
Map
Solon's for Liverpool?
Video Agent
X
V
X
A. 10
B.25
Reduce Into Unifteed Auswwer
Mapolodace
J
J
Figure 1: MapReduce Principle. Long video understanding requires both global comprehension and detailed perception without making assumptions about the videos, as shown in the example of counting large number of events. For such needs, (a) VLMs and (b) video agents are sub-optimal in terms of context lengths, sequential parallelization, and using global context information. (c) We introduce the MapReduce principle and define the Map operation dense and independent short segment perception, and the Reduce operation as global aggregation, (d) effectively achieving both inference scalability and improved performance.
(Fig. 1(b)). which harms sequence- eequence-parallel video perception, consequently, inference scalability for extremely long videos; (2) the explicit key segment retrieval contradicts global comprehension and might result in insufficient information. Take the example video in Fig. 1, for instance, the existence of 200 events breaks the basic assumption of key segment selection; and even if the model realizes the target of 200 key segments, the iterative search would cost a significant amount of time.
Our key insight to bridge detailed perception and global comprehension lies in scalably decomposing long video understanding into shorter context lengths: the model densely perceives the individual short video clips in parallel, then aggregates the condensed perception results across the whole video (Fig. I(c)). This framework is noted in how large volumes of data are handled efficiently via the MapReduce principle in distributed systems [7]: now define the Map step as parallel perception of short clips, and then the Reduce step as global aggregation for the whole video. Such native compatibility with the MapReduce principle also makes our framework friendly for inference scaling
at deployment (Fig. (d)).
Given the low cost and convenience of building video agents, we instantiate the MapReduce principle via a video agent [8, 42, 45, 53] called MR. Video. This also aligns with the trend of utilizing the foundation models in zero-shot to address challenging visual reasoning problems [6]. To unleash the capability of MapReduce, our design MR. Video introduces two critical yet previously under-explored aspects of long video understanding: (A) The captioning stage generates texts as an efficient video analysis medium, where MR. Vidco specially employs a Reduce step to provide consistent character/object names across the long videos, which benefits reasoning across long stories. (B) The analysis stage conducts question-relate advocates using question intention analysis to replace the key-segment retrieval in conventional video agents, which does not make any assumptions about the video and provides more comprehensive contexts for complicated multi-hop reasoning.
comprehension of the video. MR. Video uniquely
With both the MapReduce principle and long video agent designs, MR. Video achieves strong performance. Notably, on LVBench [39], one of the most challenging benchmarks featuring hour- long videos and diverse questions, our MR. Video achieves a more than 7% accuracy improvement over other VLMs and video agents, along with advantages several other video benchmarks.
To summarize, our contributions are:
1. We introduce the MapReduce principle from the distributed system domain to long video under- standing, offering conceptual framework that mitigates the context length, sequence-paralle! scaling, and global context limitations of previous VLMs and video agents.
We design "MR. Video." a video agent featuring multiple MapReduce stages that generate character-consistee captions and conduct innestionnnntntiin analysis, both essential for long video reasoning.
3. We highlight the strong performance of MR. Video across multiple long video benchmarks, notably represented by the challenging LVBench. These results suggest the potential of MapReduce as general principle for long video understanding.

---

2 Related Work
VLMs for Video Understanding. Existing VL.MS [3, 4, 20, 21, 22, 23, 24, 29, 30, 32, 33, 34, 35, 38, 40, 43, 44, 46, 48, 49, 50, 51, 61, 64] commonly follow LLaVA [28] by projecting image tokens to LLMs. As an image typically takes over 100 tokens in standard LLaVA model, context lengths become the major challenge for these models in long video understanding: how to digest the whole video without missing details? LongVILA [50]'s solution is increasing the context length, but inherently needs more resources and is still limited by context lengths. Another prevalent solution is decreasing the average tokens per frame via merging or pruning. Such compression can follow certain priors, e.g., similarity of features [4. 22, 32, 34, 35, 40, 46, 48, 49], or Q-former-like [13, 14, 18, 19] learnable module [23]. Notably, the recent VideoChat-Flan
[22] can support up to 10k frames with sufficient hardware. However, aggressive compression might lead to unreliable perception of visual details. Such inherent context length limitations of VLMS necessitate more flexible agentic paradigms explained below.
Video Understanding Agents. Video agents provide a meta-level LLM controller on the top of VLMs, which splits a long video into sub-tasks of short videos [8, 42, 45, 53, 60). Therefore, they are not constrained by context lengths. By imitating how humans watch videos, video agents can be treated as increasing the test- tist-time compute of VLMs via multi-round exploration [53], key-frame retrieval [8] and tool-use [42]. However, video agents still demonstrate disadvantages compared with VLMs, as mentioned in Sec. 1: (1) the sequential multi-roume exploration hinders scalability, and (2) reliance on key-frame retrieval constrains the understanding of sufficient contexts. From such aspects, MR. Video bridges these gaps with the sequence-parale Map steps and globally aggregating Reduce steps, respectively (as in Fig. (c)).
LLM Agents. Our MR. Video, in the context of long video understanding, also contributes to a broader field of research addressing complex problems with the advanced reasoning ability of LLM agents, such as software engineering [15, 52] and knowledge retrieval and reasoning [54, 55, 62]. In addition, our work aligns with the ongoing efforts to explore the zero-shot capabilities of foundation models in various visual reasoning tasks by designing the prompts without training the models, as exemplified by Visual Programming [11], ViperGPT [36], and Socratic Models [57]. With the significant accuracy improvement achieved by our MR. Video, we demonstrate that LIM agents provide an effective way to explore new frameworks at frieedla- costs.
3 Method
3.1 Overview
Although the MapReduce principle is widely applicable for handling large volumes of data with scalability, designing the concrete Map and Reduce operations for the specific task of long video un- derstanding is non-trivial. With the convenience and low costs of LLM agents, we create the prompts and workflows of VLMs to address the challenge of digesting global contexts while perceiving local details in long videos. This leads to an effective video agent: MR. Video.
MR. Video's 's 2 is in Fig. 2. It contains two MapReduce stages. (A) The "Captioning" stage (Sec. 3.2) generates dense captions, which provide concise comprehension of the video contents and serve as an efficient medium for answering multiple questions on the same video. (B) The "Analysis" stage (See, 3.3 and Sec. 3.4) conducts question-specific perception of the video. first emphasizes understanding the intention of the question (Sec. 3.3), ie.., "what the question is actually asking," " and then purposefully inspects the visual details or longer temporal spans (Sec. 3.4). The Map steps are independent and pepuuncce-prallle in both stages for different video segments, and the Reduce steps condense the segment- level results into unified video-level understanding.
Key Operations. We propose two operations that specially tailor the MapReduce for long video understanding and demonstrate beneficial behaviors unobserved by previous video agents. (1) Consistent characters/objects in captions. Instead of purely relying on captioning models, we construct workflows to improve the consistency of character names across a long video, which is beneficial to reasoning across long temporal spans. (2) Question intention analysis. We advocate combining the video contexts and questions to understand the goal of the question, such as "when, why, what," instead of relying on the key-segment retrieval adopted by previous video agents. By using thorough video contexts, our question intention analysis provides more comprehensive information.
'The displayed video is the 1st from LVBench (video link). We will consistently use it for method demon- strations for readers' convenience.
3

---

10:48
(a) Captionin
Map
spe ses lights estich hardh...
inarse trt
stech soted
samunn cefeas
spouut ur ressoves the
Redice
Menging& Renamiing Characters:
Ip [:25] spensttue [1065,
sciddrer (6054) sumanid
spessuue [B622]]
no
and.
Some Incoe
the
specin bo (childhood] tee sere steliontolllnnss
epecien nemeves
unilles sperooooo ho........
inihellnds...
incer
Specso To coos
drelchon the fee
Question 1 [Entity Recognition)
Question 2 (Reasoning)
mony sticke dosn lie protagionis put vo the incame lurner? A.3
Why doas the protogront Lie inun to her lindo? Strengthon body /
B.2
C.5
D
Dacoruttens Sapppeo Superponerr
D. Callrre
(b) Intention Anslysis
Map
This scanc in inclosst quorss withou
"Ieessed" lillkey the lighted Prenggasir
Showing yne rocenre might
speneon, migh bo the pestogosali,
somen imvinuee
The scure to tmaiieent
lrcerie daplaycd.
queetien's intertion.
sterk deplayed
Reduce
Only low socncaalion potesttal incouuiivss cnleoinniiiy thas "rsight" porppeeetinng soece (1845, taco/wher spernsr,tt placora black pet...
..laaadiics protagerioi us spersondh fo wearing conical The sccnto where pentegseom Thesearss specssor morking faggege.
sarway
(e) Goal Goal- Awwre
Map
>lw.asery" mary summrr?")
viw.osery (" wary
Showe the prroparke with weights riiing
Analysic
the summer?")
Reduce
protoqernin
yroting sugpest prosiggen cest wwwsstaaaaiiiiiibwwwwo.o attachd ho kenin, after encapit reatraiitts exggeoting Branofffi Anowerr (4)
bdlchh which corsiderd
The-thhe seeeretterd cot conoail the action of poptiin moils ints on storss bariee oction concrra Therefore. the most remoosak arswer is (Azowa]::())
Figure 2: Overview. MR. Video reformulates the MapReduce principle into three specialized stages, each designed to address a unique challenge in long video understanding. We show two types of questions focusing on visual details (left) and reasoning (right). (a) Captioning (Sec. 3.2) generates detailed captions and uniquely enhances the consistency of characters/ohjects names with the Reduce step, which is repeatedly useful for downstream analysis. (b) Question Intention Analysis (Sec. 3.3) departs from the conventional key segment retrieval adopted by previous video agents. Instead, it digests the whole question and video content to provide a comprehensive context for detailed perception. (c) Goal-Aware Analysis (Sec. 3.4) delves deep into detailed perception and reasoning over short and long temporal spans. (For clarity, MR. Video's 's intermediate texts are simplified.)
3.2 Captioning
Captions provide an efficient medium for video understanding that covers long-range contexts. Our captioning shown Fig. 2(a): (1) The Map step (See. 3.2.1) generates dense captions at the scene level independently, and (2) the Reduce step (Sec. 3.2.2) provides coherent names for repeated characters and objects for consistency. For a 2 hr video, our captioning generates 500- 2k captions for the whole video, similar to an article.
Compared with previous video agents [8, 42, 45] that rely on an off-the-shelf captioning model, we design detailed techniques to improve the captioning quality. Most notably, we optimize the framework so that every character has unique tag like "person-b" instead of general descriptions. As in Fig. 2, this enables downstream analysis to connect character across different segments.
3.2.1 Map: Dense Scene Captioning
The Map step follows a sequence- -arralell manner and generates dense captions, as in Fig. 2(a). It involves: (1) Detailed Description. We empirically discover that existing VLMs might struggle with processing video clips with signif- icant transitions. Therefore, we first prompt VLMs to check the continuity of every short clip and specify the transi- tioning frame indexes; then, we conduct the captioning task by letting the VLMs describe every continuous scene in de- tail. Such a strategy decreases existing VLMs' difficulties and provides "scenes" as atomic long video understanding units. (2) Key Characters and Objects. As preparation of consistent character names, we sparsely sample frames from a longer video segment, instruct the VL.M to iden- tify the salient characters/objects, describe
(a) Map: Extracting Characters
Chorocters from Segnentt
Choracters from Segment
conturall person
<cNidron stonding weaning rabes.
sperson parson
Reduce: Character Merging
Framess DDescrpption
Whtch characterzare the same?
commraal a> from SEG Sperson at from SRG2
Figure 3: Consistent Names. (a) The Map step extracts the salient characters/objects along with a description. (b) Then, the Reduce step uses VLM to associate the repeated characters, enhancing the consistency.

---

their identifiable properties, and specify the frame indexes most saliently showing these charac- ters/objects (as in Fig. (a)).
3.2.2 Reduce: Consistent Names
Identifying consistent characters is essential for understanding long videos. Otherwise, the analysis cannot capture the notion of "protagonist" as in the example of Fig. 2. However, a common challenge of existing VLMs is that they tend to provide a general description for a character, e.g., person, instead of referring to using a consistent name across the long video. Therefore, our Reduce step overcomes this challenge by merging the key characters extracted from the Map step, as in Fig. 3(b).
Specifically, our Reduce step decouples this task into two sub-steps: character association and caption modification. (1) MR. Video instructs the VLM to associate the repeated characters/objects by observing the salient frames of extracted characters/objeets, assigns a new set of names for every character following the format of "<entity>>> <<entit>><index>>" <innex>>" to avoid repeated names or losing semantic meanings. Finally, MR. Video accordingly updates the names in the original captions to the newly generated ones. Although using external tracking tools [8] might also be valid solution, we use VLMs because of simplicity and the fact that videos' frequently changing scenes could break the assumption of trackers.
in Fig. 3(b). (2) Then MR. Video
3.3 Analysis I: Question Intention Analysis
MR. Video emphasizes the importance of intention analysis because of the inherent ambiguity of questions in long-context understanding: the questions might only contain partial information, and the model has to recover crucial clues like "when," "how long," " and "where" in the video perceive. For example. Fig. 4 demonstrates multiple scenes potentially relevant to the questions, while only one should be correctly selected via reasoning. This stage utilizes the captions from the captioning stage (Sec. 3.2) and optionally includes video frames.
Compared with key-frame retrieval in previous video agents [8, 42] and scoring mechanisms in VLMs [12], MR. Video marks the importance of reasoning with global context to determine the relevant video segments, instead of purely relying on local video contents within short clips.
3.3.1 Map: Segment Intention Analysis
Without losing generality, we divide the video into non-overlapping short segments, each containing several atomic scenes split from the captioning stage. For hour- long video with Ik scenes, we have approx- imately 30 segments. Then, the VLM pro- cesses the segments' aggregated captions and the middle frames of each scene infer whether any scene provides helpful information for the question.
(a) Segmente
Mupe Segment Intention
Not Fitting Question's Intention (32:05- 52:14)
wolkinng towards the entrance burlding and then cuteio group paople witting and swaring orange robes. socass to motch option (A) group of mooks cross-lagee
Pieting Question's Intention (32)45-34:05
The segment explicttt desatiies sperson, bo surning Read loft seeting through windde landiccpe people sword Spttiau
Question
(e) Redurae Globae Intention
Within each segment, we instruct MR. Video to focus on "what is the question asking about" and generate a paragraph of analysis as in Fig. 2(b). Concretely, its re- sponse contains: (1) Reasoning: a para- graph analyzing the key subject/criteria mentioned by the questions and how the contents presented in the captions could align with the question in any perspective, e.g., Fig. 4(c). (2) Candidate Scenes: the LLM then lists the potential scenes that could contribute to answering the question. Please note that this is distinct from directly retrieving key frames since provides more contexts and allows frames that are helpful in indirect ways. (3) Key Subjects: The local caption segment becomes insufficient if the question mentions characters or criteria requiring global video information. So MR. Video specifies unsure criteria and their identifiable properties here for the global Reduce step to analyze.
Wher doer the
see thrrugh
[Rasaanings] need iduntifly scenss that the
vehidius ofter tsivn the veilty
protagorist looking through awinnoww zbays
Monkee sitting cronellggeed in snow
related slows or sward practice matiche suplor
Citieena shatting toguther
Warriors practicing swords
[Candadate Segmenass] (1455.4 1465.0) (20385, 2045.5
strongoot with moord fightig in the stose...
Figure 4: Question Intention Analysis. Long video questions require the model to recover the hidden in- formation of the question, e.g., who is "protagonist," what a "utility room" looks like. This motivates MR. Video's explicit understanding of the question' 's inten- tions by reasoning both the video contents and questions, instead of relying on conventional key-segment selec- tion/retrieval.
5

---

3.3.2 Reduce: Global Intention Analysis
MR. Video's Reduce step marks the key distinction with previous methods, where we reason the analyses at the video level. In principle, the Reduce step generates similar contents as the Map step but covers the contexts of the whole video. So it can localize the best scenes and subjects for the questions as in Fig. 2(b). The outputs contain the following contents. (1) Reasoning: a paragraph analyzing the key subject/criteria mentioned by the questions and how the contents presented in the captions could align with the question in any perspective, e.g., Fig. 4(c); (2) Candidate Scenes: the LLM then lists the potential scenes that could contribute to answering the question. Please note that this is distinct from directly retrieving key frames since it provides more contexts. Fig. 4(d) shows an example output of the Reduce step, which correctly discovers the relevant video scenes by figuring out the protagonist and the window.
3.3.3 Key Segment Selection/Retrieall v.s. Our Intention Analysis
Explicitly reasoning the intention of questions, i.e., completing the contexts, is significant difference between our MapReduce principle and previous video agents [8, 42, 53]. We advocate for intention analysis, combining the whole video context instead of the key segment selection, which is a critical insight of MR. Video.
Our design uses the models' reasoning abilities to inspect short video clips in detail (Map) and then comprehend the video as whole (Reduce). Although the key-segment selection of previous video agents [42, 53] and VLMs [12] implicitly refects the "intention analysis" objective by choosing few frames with the most similar features to the question,
an over-simplified model for long contexts and reasoning: in the example of Fig. 4, it is challenging to extract features reflecting "protagonist," "utility room," " or "windows" before understanding the video contexts. In addition, sequential key frame selection framework assumes the small number of key events, which is not adaptive enough for complex or challenging queries, e.g., the motivating "counting" example in Fig. I.
3.4 Analysis II: Goal-Aware Analysis
Based on the analyzed question intentions, MR. Video's final MapReduce stage purposefully gathers the information related to the questions and converts them into final answer, namely, "goal-aware analysis," as in Fig. 2(c). An essential functionality of this stage is that MR. Video should explicitly plan the rype of information it needs: attending to captions and sparse frames over longer time horizons for reasoning. e.g., Q2 in Fig. 2; or focusing on densely sampled frames benefits visual recognition, e.g., Q1 in Fig. 2. With both capabilities, our MR. Video can flexibly handle wide range of questions by aggregating the analysis.
3.4.1 Map: Goal-Aware Scene-centric Analysis
Starting from the candidate scenes generated by question intention analysis (Sec. 3.3), MR. Video proposes purposeful queries for VLMs to perceive intra-segment densely sampled frames for visual details or inter-segment sparsely sam- pled frames for global reasoning.
Candidate Scane
[Recewing]
toinn phar Ila trauala idendl andxcurt dn indrrduddisseddd thaappoll lak...
[Dwey]
der execute, connendivie, ecene):
Lateri, Card score
Lid
Candidate Scese 2 (23:16- 25:20)
acttors, whst the cequeece
"Constzering the flow of
transsorting the selin duiiry and lor does the nember of carriers change (C) 2."
Goal Proposal. When generating the queries for VLMs, we are inspired by the flexibility of "Visual Programming" [11] and ViperGPT [36]: let LLM propose its queries for the VLMs and
are in vin, cueryla, scene)
retum
Quartio:: dew poople carryting www.aadd.ooooooooo. dhraeee
C.V n.2
understand the candidate scenes in customized ways. As shown in Fig. 5, MR. Video proposes the VLM query to cover multiple aspects of the question, gathering comprehensive information.
Figure 5: Customized Queries for Perception. With this question requiring detailed visual percep- tion, MR. Video proposes goal-aware queries for the VLMs, confirming the criteria.
Perceiving Local and Global Information.
We employ the Map step as different strategies of sampling frames for VLMs to inspect. (1) Local: We densely sample frames within each short segment for objectives requiring detailed visual information, such the example in Fig. 5. (2) Global: We sparsely sample frames across different segments, e.g., the middle frames of the relevant segments identified by the intention analysis, and let VLM perceive them for information spanning longer temporal ranges. To better leverage the reasoning capabilities of LLMs, this step can also include the captions of the selected segments to simplify the perception.

---

3.4.2 Reduce: Answer Generation
The last Reduce step attends to the global context information and generates a final response. With the previous Map steps gradually summarizing the information, this Reduce step is no longer limited by context lengths and can fully unleash reasoning capabilities. As notable characteristic of this Reduce step, it merges the scene analysis results together in unified way, especially when different scenes provide contradictory perception results or require further calculation of scene-level information, such as counting queries.
3.5 Scalability Analysis
Beyond empirical accuracy, our MapReduce principle offers significant advantages from systems and computational perspective. Here, we analyze the computational costs of Lidei in comparison to peuuence-to-eqqeene VLMs and other video agents.
Premise: The Need for Comprehensive Context. A meaningful comparison of computational cost must be grounded in the shared goal of comprehensive video understanding. As demonstrated in Fig. 1 and Fig. 2, tasks in long video reasoning often require dense perception of the video's content to cover all the potential details. Therefore, our analysis is based on the assumption that all the methods are consuming the same amount of frames instead of deliberately reducing the amount of information (e.g.. VLMs using sparse frames or agents selecting few retrieved clips). Without the loss of generality, we compare the cost of our MapReduce with a conventional VLM or video agent when perceiving Nitames frames.
Comparison with Sequenne-to--Sequenc VLMs. (1) Token Count. A standard VLM must process the visual tokens from all N irames frames, resulting in a total token count of TVLM = Nirames (tokens per frame). For our MR-Video, only the initial, parallelizable captioning step processes the raw frames, consuming approximately 2 x TVLM tokens (for two passes of segmentation and caption generation). Subsequent MapReduce stages operate primarily on text or sparsely sampled frames, incurring a much smaller average cost, Tie-- Therefore, our approach expands the test- time computation in targeted manner build a comprehensive textual summary, which is more efficient for downstream reasoning. (2) Computational Cost. The advantage of MapReduce mostly lies in the computational cost. Even assuming that VLM can effectively understanding the Ntrames within its context lengths, standard transformer- based VLM has a computational complexity that quadratic with respect to the input length, i.e., Rames. In comparison, our MapReduce framework partitions the video into M parallel segments. The computation is then reduced to M x O ((Niamees/Aa ? = ONNaaess/A /M). Given that the number of segments M for long video is significantly greater than the number of 3 sequential stages in our framework, the total computational cost of MR- Video is substantially lower than that of a monolithic VLM attempting to process the same number of frames.
Comparison with Video Agents. Similar to the analysis of VLMs, we maintain the assumption that both methods start with a dense perception of Ntraess to generate high-quality captions or initial analyses. (1) Token Count. Under the dense context premise, both our method and video agents [8, 42, 45, 53] rely on an intensive initial captioning or analysis phase. Therefore, our total token counts are comparable achieve the same quality of initial understanding, (2) Critical Path and Parallelization. The primary system-level advantage of our MapReduce principle is its ability to shorten the "critical path" of inference, enabling superior scalability. Consider the counting task in Fig. reffig:teaser, where over 50 key events of a soccer video must be identified.
(a) A video agent relying on iterative, sequential key-frame retrieval would have a critical path of over 50 steps, with its length varying unpredictably based on video complexity and reasoning depth.
(b) In contrast, MR- Video executes its plan using parallel "Map" steps, resulting in a short and controllable critical path of just 3 MapReduce stages. This inherent parallelism means that the video processing throughput can scale linearly with the number of available GPUs or VLM inference endpoints, a crucial advantage for practical deployment.
4 Experiments
4.1 Datasets
Evaluation Dataset Selection. To validate the MapReduce principle within our limited budget, we focus on the challenging long video benchmark: LVBench [39]. Compared with others [9, 31,35, 63), I. B.Veench features more extremely long video durations and challenging questions, as directly reflected by the lower accuracies of oftc-offtheeart models. With a limited budget, we expand the breadth of
7

---

Model
ER
BU
KIR
TG
RE
SUM
Overall
Proprietary VLMs
Gomini-1.554 [37]
32.1
30.9
39.3
31.8
27.0
32.8
33.1
GPT4o [1]
48.9
49.5
48.1
40.9
50.3
50.0
48.9
Gemini-2.0-Fian [37]
47.4
48.5
56.8
39.3
44.4
41.4
48.6
Open-sourced WLMs
IntemVL2-40B [6]
39.7
42.5
41.4
TimeMarker [5]
42.8
39.1
34.9
38.7
38.2
48.8
41.3
Owen2-V1--225 [38]
38.0
41.1
38.3
41.4
46.5
46.6
41.3
VideoLaMA3-2B [58]
41.5
39.7
44.0
32.7
45.8
25.9
41.6
mPL.UG-OOW1 [56]
46.0
41.6
40.4
43.5
InternVL.2.5-788B [6]
43.8
42.0
42.1
36.8
51.0
37.9
43.6
VideoLLaMMA3-78 [59]
45.8
42.4
47.8
35.9
45.8
36.2
45.3
Qwen2.5-VI.72BB [2]
47.7
ReTake [40]
46.2
45.0
45.8
27.6
47.8
VideoChat-Flash [22]
51.1
46.0
49.0
38.9
48.5
34.5
48.2
GI.M-4V-Plus [10]
46.2
47.8
54,1
42.7
46.5
37.9
48.7
AdaReTaaee [41]
53.0
50.7
62.2
45.5
54.7
37.9
53.3
Video Agents
VideoAgent [42]
28.0
30.3
28.0
29.3
28.0
36.4
29.3
VideoTree [45]
30.3
25.1
26.5
27.7
31.9
25.5
28.8
VCA [53]
43.7
40.7
37.8
38.0
46.2
27.3
41.3
MR. Video (Ours)
59.8
57.4
71.4
58.8
57.7
50.0
60.8
Table 1: L.Bench Comparison. Our MR. Video significantly outperforms previous methods by large >7% margin, suggesting the effectiveness of the MapReduce principle. The VLM accuracies from the official leaderboard as of 5/10/2025, and the video agent accuracies from VCA [53]. The columns from "ER" to "SUM" represents different question types LVBench, such as "entity recognition" and "summarization," details are in the supplementary materials.
evaluation using the subsets other representative video understanding benchmarks, especially the long video parts of LongVideoBench [47], Video-MME [9], and EgoShema [31].
Dataset Settings. LVbench [39] curates 1,549 questions on 103 videos ranging from 30 min to 2 hrs, covering 6 video categories. We utilize the LVBench data as follows. (a) As of May 15th 2025, out of 103 videos are unavailable from YouTubb for downloading. So, our comparison in Sec. 4.3 utilizes all the remaining 1,492 questions. (b) For the ablation study (Sec. 4.4), we form subset save the budget by selecting the first video of each video category in L.Vencc. This subset has vidcos and 98 questions in total. For additional evaluation, LongVideo VideoBench's validation set, (2) the long vidco subset VideoMME without subtitles, and (3) the validation set of EgoSchema. The evaluation metrics are all accuracy for the benchmarks. More details on the datasets are the Sec. D.4.
(1) the longest subset of
4.2
Implementation Details
MR. Video Details. Our MR. Video demonstrates a simple framework validating the MapReduce principle, only requiring one LLM for text understanding and one VLM for image understanding. To save our expenses, we utilize Gemini-2.0-Flash [37] as our VLM, and we only use GPT4o to process texts. On average, generating the dense captions for an hour-long video requires approximately $0.8 of Seemini-200-Flaass and answering each question from LVbench costs SO.4 GPT4o on average. We provide further details, especially the prompts, Sec.
Controlled Context Lengths. We highlight a vital implementation detail so that our video agent is meaningful for overcoming the context length challenges: we explicitly control the VLM perceive less than 40 frames per query, significantly less than the typical 256 or even more frames for long video VLMs [22]. This ensures MR. Video does violate the motivation of building video agents.
Baseline Evaluation. Because of the high cost of evaluating models, we mainly refer to the numbers on the leaderboards provided by the authors in comparison (Table I and Table 2). The only exceptions are: (1) For VideoAgent [42] and VideoTree [45] on Long VongeideeBench (Table 2), we their open-source code, the GPT4o model, and our captions for fair comparison; (2) For base VI.M Gemini-2.0-Flash we follow the standard VL.M setting by uniformly sampling 256 video frames per video. More details are in Sec. D.5.1.

---

4.3 Main Comparison
4.3.1 LVBench Comparison
By tailoring long video understanding insights into the MapReduce principle. our MR. Video demonstrates significant advantage the challenging LVBench in Table 1. Using the cheap Gemini-2.0-Flass and a smaller context length, our MR. Video improves the base VLMs and all the previous video agents primarily using better GPT4o. Therefore, such comparison suggests the effectiveness of our MR. Video for long video understanding.
4.3.2 Breadth Comparison
As shown in Table (LVBencc performance is listed for reference), MR. Video demonstrates significant advantages on the long video benchmarks than the previ- ous video agents, despite using a cheaper VLM Gemini-Flash. MR.
Boachmarks
LVBench LongVidto Overall 4101s
Val (Lang)
EguSchema. Val
Video-MME Long (wlo Suh) 2386s
Average Duration
1434s
VLMS
Video also consistently outperforms the base VLM with a smaller con- text length, while the previous video agents commonly underperform their VLM, GPT40. Therefore, this indi- cates the effectiveness of MR. Video and the significance of the underlying MapReduce principle for long video understanding.
GPT4o [1]
48.9
58.6
70.4
65.3
Gemini-2.0-Fllan [37]
48.6
45.7
71.2
63.0
Video Ageary
VidenAgent [42] Video [45]
29.3
28.8
39.2
67.0
53.1
VCA [53]
41.3
73.6
56.3
MR. video (Ours)
60.8
61.6
73.8
63.4
Table Breadth Comparison. MR. Video performs better than other video agents. More importantly, we consistently outperform the base VLM, Gemini-2.0-FFan context length, while other video agents commonly under- perform their VLM, GPT4o. (Long VideoBench accuracy of GPT40 is from their paper, EgoSchema accuracies are from VCA [53], and Video-MME accuracies are from the official leaderboard and VCA's paper [53].)
with smaller
To guide the future analysis of video agents, we also notice the dis- tinct question styles of LVBench, Long VideoBenh,, and VideoMME, leading different scales of advan- tage between our video agent and the base VLM. Please refer to our discussion the Sec. D.4.
4.4 Ablation Study and Analysis
We utilize the LVBench subset (explained in Sec. 4. to analyze our Map and Redyce operators.
Consistent Character Names in Captions. Following the order of the MapReduce steps, first analyze the benefits of the Redyce step in captioning (Sec. 3.2): providing consistent charac- ters/objects names. As shown in Fig. 2, such consistent names enable the analysis capture coherent behaviors of characters. Without consistent names, we observe a significant performance drop ("w/o Consistent Characters" in Fig. 6).
Question Intention Analysis. understanding the video contexts instead of the key-segment se- lection used by previous video agents. To analyze their differ- ences, we utilize the target video clips annotated by LVBench to assess whether intention analy- sis can better localize the key segment: (I) whether the candi- date scenes selected by our ques- tion intention analysis overlap with the annotated target clips; (2) whether the intention analysis is better than retrieving the key scene matching the embeddings of video clips and questions.
As clarified in Sec. 3.3.3, we advocate comprehensively
(a) Ablation Study
(b) Key Frame Recall
7114
MR. Video
63.
weo Comsistent Clarractess
31.1
w/o Goal- Analysis
$2,0
0.0
Video
MM4-mmbed
Figure 6: Analysis. (a) We investigate the benefits of MR. Video components. (b) The comparison between our question intention analysis and the key frame retrieval suggests the necessity of com- bining more video contexts for localizing the critical information.

---

Michail
Queddia
Quection
liweesacy Nt paakk orn vide?AA
0.35
C200
Riaredoo Magaary took www. A B. H.Hull
Paa D. Pool
Inication, Analysaa (Map)
Intentioen Amatrsia (Map)
1:37 gual spkget.s
to deatting by splgerr.ar ar
elltintion by splepocse
1pa satuer fore of skggass) als knowe Mikkell.
batk brank lac. Stnue lua
Ue
Intention Analysi (Resace)
Uypec pas
bakw total
speses wherefh the ball isakury sykger in, fnon al by cosnnng cheie scasts - the ant sunthre green
felowed edetiition
Geal-hecree Analyda (Redace)
eaggecting rag whure be fastially ansind" Chamation).. (Nithhdd sperssn br).
[L Reaerrisa Segenart elter 22.45 support wutting ruch as domrite acuasa is ariia (t
Intal samberr goou mestianed snalys waich dosa retrnacc opt00. [Arowca): 2.2. 222.. igenalune paou band asalyir... -)
Figure 7: Case Analysis. (Left) Recalling the motivating example (Fig. 1), MR. Video checks every scene in detail (Map) and aggregates the whole video (Reduce). Although it misses some goals due to striet criteria (shooting, goal, and celebration), MR. Video shows the desired behavior of counting exhaustively. (Right) This example demonstrates how a consistent character name (Mikhail) benefits the reasoning process of MR. Video (first night).
First, MR. Video correctly localizes the relevant scenes for 70.4% of the questions, where video typically contains 500-2k scenes. Second, we employ MM-Embed [25], a state-of-the-art multi-modal retrieval model, to conduct the key-frame retrieval. Under a fair comparison setup (details in the Sec. D.5.2), retrieval achieves an accuracy of 34.4%, which is significantly worse than question intention analysis (Fig. 6(b)). This suggests the necessity of question intention analysis, which combines global context for localizing the key video segments.
Goal-aware Analysis. Goal-aware analysis (Sec. 3.4) provides the video agents with opportunities to delve deep into video content after coarse analysis. Without this final MapReduce stage, the performance drops significantly in "w/o Goal-Aware Analysis" (Fig. 6(a)).
4.5 Case Analysis
Finally, we closely observe the behavior of MR. Video and find it demonstrating a successful long video understanding process. (1) We recall our motivating problem of the challenging counting question: as in Fig. 7 (left), MR. Video indeed shows the behavior of exhaustively perceiving each video clip, checking the criterion, and summing up the numbers. Although its number is smaller than the ground truth due to strict checking criteria, MR. Video shows valid path towards addressing large number of events in long videos. (2) In this travel video, MR. Video demonstrates the multi-hop reasoning benefit from consistent names and explicit analysis of the event orders from global contexts, which are crucial premises for addressing complicated video reasoning.
Conclusion
To address the challenge of understanding both local details and global contexts in long video understanding, we introduce the MapReduce principle and formally define its operations in MR. Video. Compared with previous VLMs and video agents. MR. Video shows the advantage in smaller context length, better sequence-parallelism and inference scalability, and comprehensive global context understanding. Targeting the under-explored challenges of long videos, we further propose consistent character names in captions and question intention analysis to replace the conventional key frame retrieval. Finally, MR. Video achieves significant advantage on multiple long video benchmarks, showing the potential and effectiveness of MapReduce.
Limitations and Future Work. (1) We utilize the LLM agents paradigm because of its low cost, but the MapReduce principle is also conceptually compatible with VLMs, where local attention compresses short video segments and global attention at the final layers aggregates the global contexts. Therefore, a potential future work is to formulate and verify MapReduce for VLMs. (2) Another limitation of LLM agents is that L.MS are not aligned with the video understanding. especially when the texts used for visual reasoning could lose nuanced visual information (analysis supplementary materials), another future work is to conduct post-training for the LLMs of the video agents.
10

---

Acknowledgments
This work was supported in part by NSF under Grants 2106825 and 2519216, the ONR Grant N00014- 26-1-2099, the DARPA Young Faculty Award, the NIFA Award 2020-67021-32799, the Amazon- Illinois Center on AI for Interactive Conversational Experiences, the Toyota Research Institute, and the IBM-Illinois Discovery Accelerator Institute. This work used computational resources, including the NCSA Delta and DeltaAl supercomputers through allocations CIS230012, CIS240133, and CIS240387 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, as well as the TACC Frontera supercomputer, Amazon Web Services (AWS), and OpenAl API through the National Artificial Intelligence Research Resource (NAIRR) Pilot.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv preprint arXiv:2303.08774, 2023. 8,9
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-VE VL technical report. arXiv preprint 2502. 2025. 8
[3] Wenhao Chai, Enxin Song, Yilun Du, Chenlin Meng, Vashisht Madhavan, Omer Bar-Tal, Jeng- Neng Hwang, Saining Xie, and Christopher D Manning. AuroraCap: Efficient, performant video detailed captioning and a new benchmark. arXiv preprint ar XrIiv:2410.0305,, 2024. 3
[4] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In ECCV, 2024. 3
[5] Shimin Chen, Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma. Timemarker: A versatile video-Ilm for long and short video understanding with superior temporal localization ability. arXiv preprint or X 1. 182/ 11, 2024. 8
[6] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, et al. InterVV:: Scaling up vision foundation models and aligning for generic visual- linguissi tasks. CVPR, 2024.
[7] Jeffrey Dean and Sanjay Ghemawat. MapReduce: simplified data processing on large clusters. Communications of the ACM, 51(1):007-113, 2008. 2
[8] Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li. Zhi Gao, and Qing Li. VideoAgent: A memory- augmented multimodal agent for video understanding. In ECCV, 2024. 1, 3, 4, 5, 6,
7
[9] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis. arXiv preprint arXiv:2405.21075. 2024. 7, 8, 22, 32
[10] Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. ChatGLM: A family of large language models from GLM-130B to GLM-4 all tools. arXiv preprint arXiv:2406. 12t.222466..79,,, 2024. 8
[11] Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In CVPR, 2023. 2, 6
[12] De-An Huang, Subhashree Radhakrishnan, Zhiding Yu, and Jan Kautz. Frag: Frame selec- tion augmented generation for long video and long document understanding. arXiv preprint
arXiv:2504.17447, 2025. 6,
[13] Andrew Jaegle, Sebastian Borgeaud, Jean-Baptiste Alayrac Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver 10: A general architecture for structured inputs & outputs. In ICML, 2022.
11

---

[14] Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. ICML, 2021. 3
[15] Carlos Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-Bench: Can language models resolve real-world github issues? In ICLR, 2024.
[16] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, al. Wiva-OOneViiinn Easy visual task transfer. arXiv preprint 2oXiv:2408.033260 2024. 1
[17] Feng Li. Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. -LaVA-NeXT-Interleaae Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint arXiv:2407.07885, 2024.
[18] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In ICML, 2023. 3
[19] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision-language understanding and generation. ICMI., 2022. 3
[20] KunChang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. VideoChat: Chat-centric video understanding. arXiv preprint arX5v:2305.06355, 2023. 3
[21] Rui Li. Xiaohan Wang, Yuhui Zhang, Zeyu Wang, and Serena Yeung-Levy, Temporal preference optimization for long-form video understanding. arXiv preprint arXiv:2501.13919 2025.
[22] Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang. Jianfei Gao, Kun- chang Li, Yinan He, Chenting Wang. Yu Qiao, Yali Wang, and Limin Wang. VideoChat-Flash Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.0055 2024. 3, 8
[23] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-VID: An image is worth tokens in large language models. In ECCV, 2024.
[24] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-LLaVA: Learning united visual representation by alignment before projection. In ACL, 2024. 1,
[25] Sheng-Chieh Lin, Chankyu Lee, Mohammad Shoeybi, Jimmy Lin, Bryan Catanzaro, and Wei Ping. MM-Embed: Universal multimodal retrieval with multimodal LLMs. In ICLR, 2025. 10, 33
[26] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, arXiv preprint 21100:231, 2023.
(27) Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLxVA-NeXT: Improved reasoning, and world knowledge, January 2024.
[28] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023. 1,3
[29] Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi. Shiyi Cao, Yuxian Gu, Dacheng Li, et al. NVILA: Efficient frontier visual language models. arXiv preprini 24X1:22412.04448,, 2024. 3
[30] Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx MLLM: On-demand seatial-temporal understanding at arbitrary resolution. In ICLR, 2025. 3
[31] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. EgoSchema: A diagnostic benchmark for very long-form video language understanding. In NeurIPS, 2023. 8, 32
[32] Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. TimeChat: A time-sensitive multimodal large language model for long video understanding. In CVPR, 2024. 3
12

---

[33] Yuzhang Shang, Mu Cai, Bingxin Xu, Yong Jae Lee, and Yan Yan. LLaVA-PruMerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.1538, 2024.
3
[34] Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, al. LongVU: Spa- tiotemporal adaptive compression for long video-language understanding. arXiv preprint orXiv:2410.17434 2024.
[35] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, et al. MovieChat: From dense token to sparse memory for long video understanding. In CVPR, 2024. 3,7
[36] Didac Surís, Sachit Menon, and Carl Vondrick. ViperGPT: Visual inference via python execution for reasoning. In ICCV, 2023. 2, 3, 6
[37] Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean- Saan-Bppistte Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312./180, 2023. 8,9
[38] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-VL: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint axXiv:2409.12111, 2024. 3,8
[39] Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. LVBench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406. 200805300 2024. 2, 8, 22, 32
[40] Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao. and Liqiang Nie. ReTaKe: Re- ducing temporal and knowledge redundancy for long video understanding. arXiv preprint
arXiv:2412.20504, 2024. 3,8
[41] Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nie. Adaretake: Adaptive redundancy reduction to perceive longer for lallo-lan understanding. arXiv preprint arXiv:2503. 12559, 2025. 8
[42] Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. VideoAgent: Long-form video understanding with large language model as agent. In ECCV, 2024. 1, 2, 3, 4, 5, 6, 8, 9, 23, 33
[43] Xidong Wang, Dingjie Song, Shunian Chen, Chen Zhang, and Benyou Wang. LongLLaVA: Scaling multi-modal L.MMs 1000 images efficiently via hybrid architecture. arXiv preprint MxXi:2449.002889 2024. 3
[44] Yi Wang, Xinhao Li, Ziang Yan, Yinan He, Jiashuo Yu, Xiangyu Zeng, Chenting Wang, Changlian Ma, Haian Huang, Jianfei Gao, et al. Intern Video22: Empowering video MLLMs with long and rich context modeling. arXiv preprint arXiv:2501.123888 2025.
[45] Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jaehong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. VideoTree: Adaptive tree-based video representation for LLM reasoning on long videos. arXiv preprint arXiv:2405. 19209, 2024. 1, 2, 3, 4, 7, 8,
[46] Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. LongVLM: Efficient long video understanding via large language models. In ECCV, 2024.
[47] Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. In NeurIPS, 2024. 8, 22,
[48] Long Xing, Qidong Huang, Xiaoyi Dong, Jiajic Lu, Pan Zhang, Yuhang Zang, Yuhang Cao. Conghui He, Jiaqi Wang. Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. In CVPR, 2025.
[49] Mingze Xu, Mingfei Gao, Zhe Gan, Hong- You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. Slowfast-llava: A strong training-free baseline for video large language models. arXiv preprint arXi::2407118811, 2024. 3
13

---

[50] Fuzhao Xue, Yukang Chen, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang. Haotian Tang. Shang Yang, Zhijian Liu, et al. LongVILA: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 10188, 2024.
[51] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2.5 technical report. arXiv preprint WrXi::2412.15115 2024. 3
[52] John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-Agent: Agent-computer interfaces enable automated software engineering. In NeurlPS, 2025. 3
[53] Zeyuan Yang, Delin Chen, Xueyang Yu, Maohao Shen, and Chuang Gan. VCA: Video curious agent for long video understanding. arXiv preprint mrXi:22412.00471, 2024. 1, 2, 3, 6, 7, 8, 32
[54] Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhut- dinov, and Christopher D Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. arXiv preprint wrXiv:180.006600 2018. 3
[55] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. In ICLR, 2023. 3
[56] Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, and Jingren Zhou. mPLJUGOWLL3: Towards long image-sequence understanding in multi-modal large language models. In ICLR, 2024.
[57] Andy Zeng. Maria Attarian, Krzysztof Marcin Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael S Ryoo, Vikas Sindhwani, Johnny Lee, et al. Socratic models:
Composing zero-shot multimodal reasoning with language. ICI.R 2022.
[58] Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng. Yuming Jiang, Hang Zhang, Xin Li, et al. VideoLLaMA 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint arXiv:2501.13106 2025.
8
[59] Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang. Hang Zhang, Xin Li, Peng Jin, Wenqi Zhang, Fan Wang, Lidong Bing, and Deli Zhao. Videollama 3: Frontier multimodal foundation models for image and video understanding, arXiv preprint arXiv:2501.13106 2025. 8
[60] Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple LLM framework for long-range video question-answeriig In
EMNLP, 2024. 3
[61] Yuanhan Zhang, Bo Li, Haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. LLaVA-NeXT: A strong zero-shot video understanding model, April 2024.
3,3
[62] Andy Zhou, Kai Yan, Michal Hilapenttok-Rothmaas Haohan Wang, and Yu-Xiong Wang. Language agent tree search unifies reasoning acting and planning in language models. In ICML, 2024.3
[63] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. MLVU: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04222 2024. 7
[64] Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Fstruch, Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An exploration of vidco understanding in large multimodal models. arXiv preprint arX1v:2412.103600, 2024. 3
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction accurately reflect our contribution of introducing the MapReduce principle for long video understanding and building an effective video agent MR. Video with significant improvement.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations at the end of the paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification:
This paper addresses the practical problem of long video understanding and does not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the implementation details in both of the main paper and supple- mentary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have provided the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. ca/
Srriides/CodeSummissttt
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/nips.cc/publlli httpcccccppcccccecceeeec.cc gwwc///gwge///eeeeeeeeeee http://ggguieee/eeeuuuuutt ssi onnolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have included the details of our datasets, evaluation, models, and prompts in the main paper and supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to the high cost of evaluation (more than Ik dollars per run), we follow the standard practice of video agents in running the evaluation once.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have clearly analyzed the costs of our models and APIs in the implementa- tion details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't I make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / /ttpr//eeruppeeeeee//ee cc/ ceQuidelineds
Answer: [Yes]
Justification: This project does not violate the ethics code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
This work has no negative societal impact by proposing a long video under- standing framework. However, the usage of large language models for agents might carry the original bias of the Gemini or GPT models. We clarify such impacts the supplementary materials.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not have such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We properly cite and credit the datasets and models used in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. www.coo////oo//o datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We have not introduced new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Our research does not involve human subjects crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our research does involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
Justification: We have clarified using Gemini and GPT as the essential backbone of our model.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https fattps://nette weweurrpps.cc/ooffeeeeeeeee for what should or should not be described.
21

---

A Delving into Long Video Benchmarks
In Sec. 4.3.2, we mentioned the fact that the long video datasets have nuanced differences in their preferred long video understanding capabilities, so the improvement and pattern of MR. Video displays different margins of advantage on these datasets. For instance, the improvement of MR. Video over the base VLM on LVBench [39] and LongeBdeBencch [47] is more significant than Video-MME [9], and such observations reveal the common challenges of video agents. In this section, present several representative examples showing such distinctions of datasets and discuss the advantages and challenges of video agents. Please note that all of these benchmarks have curated a diverse set of questions. We demonstrate examples only provide intuition of the complexity of question styles instead of claiming that these benchmarks can be solved with a few techniques.
We show the examples in Fig. A, including the representative questions from LVBench [39], LongVideo VideoBench [47], and Video-MME [9].
LVBench
IO
42.50
50:56
W caew on the grord
W disces
Wath hondlerckitiin
(b) LangVideoRench
Emission
Rellcction
Emission
Rollection
pupacc rettie shere cueblne derign canter wed Trstulee Jeton, whith of ralba comartor appaored
Tea rnd ho agrredd thore is Claste of (itg light ceertting strnng light the centon
Black right dry there apple cinuliar oigeet mviting pipple light seewerdh "Pockyllarrrr Nebeld issothed on fheright
ladigrend
desiga and sedeeeothheeee
rodkiing linggng
design
Morked
a blue zackgrousd saadd the curtor a spar dling wita domudy paciad dati
(<) Video-MME
cing wwgguthhonn inhiiceges
Elesse relictten lecw thic of
B. Receapetseed bis oecadciine with dhe hardwark liscolaaggss.
years.
Figure A: Examples of Different Long Video Benchmarks. The benchmarks demonstrate different nuanced styles and desired capabilities from the long video understanding models. (a) LVBench [39] requires the model to precisely localize the key information by understanding the story, capturing the characters, e.g., protagonist, and comprehending the question. (b) Long VideoBench [47] also emphasizes the importance of finding the key information, but the query is more explicit by directly naming the property to search for. (c) Video-MME [9] shows the questions closer to the style of interpretive queries, requiring the models to have rough speculation and summarization of the video.
LVBench. For LVBench, the model has to localize the scene of "solo fight" correctly and understand the meaning of "knock down" and "wipe face" answer the question. Notably, the model has to integrate the contexts of the video and speculate the "protagonist" first to execute this task.
Long VidcoBench. Although both require precise localization, Long VideoBench is different from I.Vench. LongVideo VideoBench provides explicit and accurate visual cues for the model to localize the object, but the model has to propagate such information across the temporal axis to answer the
22

---

question. Compared with LVBench, Long VideoBench emphasizes models' visual detail perception and temporal association abilities.
Video-MME.
Unlike the above two benchmarks, many questions in Video-MME are not about specific event. Instead, they are more interpretative, similar to the impression of human after watching the videos.
Comparatively, LVBench and Long VideoBench emphasize the challenges of localizing or multi- ple key video clips and exact matching contents, while Video-MME contains more interpretative questions similar to how humans gain intuitive impression of a video segment. Even LVBench and Long VideoBench are slightly different: LongVideote VongVideBench provides more explicit vision-centric cues, and LVBench specifies more from story event aspect.
The improvement of our video agent over the base VI.M, especially the improvement on LVBench and Long VideoBench, requiring the precise localization of information, demonstrates the advantage of video agents in localizing critical visual information by reasoning about the overall video context. Comparatively, video agents relying on text-based reasoning might lose visual details, making them less effective for interpretive questions like Video-MME, Even so, our MR. Video still outperforms the base VLM with smaller context length, while all the other video agents fall behind their base VLMs. Therefore, the above analysis indicates the necessity of the MapReduce principle in handling wide range of video tasks compared with other video agents.
B Scalability Analysis
In this section, we provide a detailed analysis of the token consumption of our Video frame- work, accompanying Sec. 3.5. We compare our method with representative open-source baseline, VideoAgent [42], on the LongVideo
benchmark, following the setup in the main paper. This analysis reveals how the MapReduce principle intentionally utilizes more tokens to achieve more comprehensive and reliable understanding of long videos, and how this design leads to superior inference-time time scalability.
As shown in Table A, starting from identical video captions, our MR-Video consumes approximately 14x more tokens than VideoAgent achieve significantly higher accuracy. This substantial difference in token usage is not an incidental byproduct but a deliberate design choice central to our framework's philosophy. While VideoAgent restricts its agent maximum of 4 rounds of interaction with the video, our approach requires the agent to densely perceive the entire video content.
Method
Avg Input Tokens Per QA
Avg Output Tokens Per QA
Accuracy (%)
VideoAgent [42] MR-Video (Ours)
7.695
383
47.6
109,522
4,908
61.6
Table A: Token consumption and accuracy comparison on LongVideoBenchl Our MR- Video inten- tionally consumes more tokens to densely perceive the entire video, leading to significantly higher accuracy.
Such a contrast directly reflects the advantage of our design in densely perceiving the video, which is necessary (as explained in Sec. 3.5). More importantly, simply increasing the token budget does not trivially lead to better performance. In fact, VideoAgent's 's own ablation study (Fig. 3, left in their paper) suggests that increasing the number of perception rounds even decrease. This observation motivated our exploration of a new scaling paradigm. Instead of pursuing greater depth (more rounds of searching for key frames), our MapReduce principle improves the breadth of understanding by optimizing for information coverage.
cause performance to saturate or
At first glance, it may seem contradictory that method requiring more tokens can offer better inference-time scalability. The key to understanding this is to analyze the critical path of computation when serving the model. Consider the event-counting scenario from Fig. 1, where over 50 goals must be identified. (1) VideoAgent relies on an iterative, sequential process of key-frame retrieval. Its critical path would consist of more than 50 sequential steps to identify all the key events, which grows
23

---

19:45
20:066
20:13
Analysts
motter foreigners illegaa weapons
diy?
Bribery supparted persan is paying quard with Goarth andthey upret... (ocblio pane could messon... likaly seswar
must
pass
26
26:16
26:65
Question
Analysis
Whottssppen
éhnons thnens-tnees?
The 26-43 26.54,
A child rides hur buck
kimono seeeling
mun linelse breide
desidi her child with displayed the [S)
talles
child hite her with muslless
Figure B: Failure Case Analysis. MR. Video's failure largely comes from (1) the VLMs fail to understand the narratives and scene transitions (example I); and (2) VLMs fail to capture visual details (example 2). (Noodles and the blurry face are pointed with arrows.)
with video complexity or reasoning hops (2) MR-Video, however, relies on highly parallelizable "Map" steps. This results in controllable critical path of just 3 MapReduce stages.
Consequently, our MapReduce principle is highly amenable parallel computation. The system's throughput (i.e., the duration of video processed per unit of time) can scale linearly with the number of available GPUs or VI.M inference endpoints. This design makes MR-Video exceptionally well-suited for practical, large-scale deployment where latency and throughput are critical.
C Failure Case Analysis
As the video agents utilize VLMs to interpret the video contents, the performance of the video agents is constrained by the underlying VLMs and LLMs, especially when the information relevant the question is already localized successfully by the video agent. (1) The underlying VLM fails to capture the nuanced storyline in the example of Fig. B: even though the model notices the bribery behaviors, it fails to conclude the correct answer due to not understanding the narrative of the vidcos. (2) Moreover, our video agent cannot recover the visual details overlooked by the VLM, such the noodle and the woman's s face in example Fig. B.
D Prompts and Implementation Details
D.1 Captioning Prompts
We describe the detailed steps and prompts for our dense captioning of the video (Sec. 3.2). All the datasets share the same captioning prompts.
Map: Dense Scene Captioning. As in Sec. 3.2, we let each short video segment produce its dense captioning, involving the following three map steps all the video segments are independent within each step to support parallel inference:
1. We split each 10s video segment into individual scenes and check if the first scene of segment can be merged with the last frame of the previous segment. Scene splitting prompts are in Table B, and the "Scene Merging" prompts are in Table C.
2. We identify the salient characters and use them to generate the dense captions in each video segment. The prompts in Table D.
3. With the selected characters, we generate the dense captions each scene with the prompts in Table E.

---

*a Context
You given continuoue screeushota video correeponding approxinately seconda of video duration, and provide detailed, faithful, and accurate analysie of this video segnent. The segments based
objective
analysin group the video of captioning and question answering.
for
Instructions
To perform the analyais of decomposing, video into shorter parta, lat's do it step by step. provided freues of seguent briefly and accurately. You should cover each action and event in the clip. The description accurate.
Based
seguent, please deecribe contente
"[1. Description] the following question: "Is this video seguent single scene or a combination of maltiple scenes?" The definition of a scene is single, self-contained, and that could be easily sumaarized with a header: "[2. Single:]" ansver
should be detailed, faithful,
Based your description, please
continuous
hunan.
should
previcus question is "be", please provide index frane(s) separating scenes from the given frame. Your answer should come with header: [3.. Franes]: and in the format intagers.
*# Example
responae should
following
[1.
Description] : This video shows
Single: yes/no]:
[3. Frames]: [6,
Please pay special attention
The praciae localization of the frames in very important for downatrean tasks.
should be consistent with the number of scenes should be one more than the number of fremes in the list. If you provide 0 franes give cunnary; you provide frame,
The sumarization
you provided. instance,
images display should be 2 aumariies; if provide frames, there ahould be sumnaries, etc. Now you will be presented the video franes, please perforn the analysis carefully.
scene, you
Table B: "Seene Splitting" prompts the Captioning stage (Sec. D.I).
You are going help with deternining if short video segnent is consistent scene. You will given a few continuous screenehots the video clip, and provide detailed, faithful, and accurate analysis of this videa segnent.
objective simple: if with *the first frane*. Answer with "yes" or "no".
clip starting
frane is
Now you will be presented the video frames, please perforn the analysis carefully.
Table C: "Scene Merging" prompts at the Captioning stage (Sec. D.1).
Reduce: Consistent Characters and Objects. As in Sec. 3.2, our additional "Reduce" step enhances consistency by merging the repeated characters into unified names. involves the following steps:
1. We iteratively check if the characters from two video segments overlap with the prompts in Table F.
2. After assigning new names to all the characters/objects, we modify the old names in the original dense captions with the prompts of Table G.
D.2 Analysis Prompts
describe the prompts for question intention analysis (Sec. 3.3).
Map: Segment Intention Analysis. We let standard LLM check the scene-level information and understand the user intentions. Each chunk of captions contains 32 scenes. Its prompts are in Table H.
Reduce: Global Intention Analysis. This step utilizes LLM to process the segment-level analyses from the previous step and unify them into a condensed video-level analysis. The prompts in Table The most critical part is explicitly instructing the LLM conduct video-level reasoning and find the most proper scenes.
D.3 Analysis II Prompts
This section provides the details and prompts for MR. Video's goal-aware analysis (Sec. 3.4).
25

---

Notivation
emooth You are paritipating in video captioning task, but you can only watch a few franes of the video and lack brosder context. Therelore, you use using character-qentric and object-centzic visual nenory that stores the key characters and objecte in the video. Your objective is to identify the rideo,
potential key characters and objects visual nemory for dounstream tasks.
influential, organize them into
# Input and Output
You will be given the following imputs:
Input
You vill have everal sparsely sampled frames of the current video clip.
Output
Your output will have the folloving format:
[1. Appeared Characters]; You will return liet the nanes of the charactere and objects that memory. Strictly follovs fornat: DIAMEI, NAME2,
appeared
....
return scene. Each item should contain the name of the character, representative frame of the character, daacription about. how identify the character
Character Details]:
agpeared
current
frane. ia:
[Visual Nenory 1:] [[HAME: nane], IDESCRIPTION: descriptionl, [FRAXE: index of the selected frame to display this character]] [Visual Memory Ende] [Visual Nenory 2:] [[NAME: namel, [DESCRIPTION: description], [FRAME: index selected
display this cbaracter]] [Visual Hemory Ends] Guidelines:
1. HAME should be general name, such person_a, person_b, person_c, object_a, etc. Try to be rigoroua and faithful to the video without naking assumptiona.
DESCRIPTIOR should be short description of the character's and object's appearance and properties, especially hov to uniquely identify the character or object from the representative frame. reprecente cheracter object scene,
FRAME should the index favorably the nost salient frame shoving the front face of the character, It should start from O.
## 2.3 Example Dutput:
[1. Appeared Characters]: ["person_a,, "peraon_b"" b", "dog_a"]
[2. Character Detaile]:
[Visual Nemory 1:]
[[MME:: person_b), [DESCRIPTION: man with short hair and glasses in the frame),
10]] [Visual Nenory
person_c], [DESCRPPTIO:: voman with long hair and blue dress in the franc], [PRAME: 20]] [Visual Mamory Ends]
(Visual Nemory 2:]
[[MME::
[[NAME: dog_al, [DESCRIPTION: with standing beside the man with ahort hair], [FRAME: 10]] [Vieual Nemory Ends]
[Viaual Nemory 3:]
# 3. Guidelines
an easy task, please sure use your advanced reasoning sbility and step carefully. The following guidelines are very important for you to finich this task:
every iten
video, trying perceiva the video and understanding the deeper plots of the video.
Please imagine yourself as hunan satching
things
When you are selecting the characters for the vieual memory, please be picky:
(a) Only select the characters and objects that you believe salient and could significantly
influence the plat. It could be person in the novie, an animal in the documentary or cartoon, etc. judgenents.
your
(b) Only includa character if it is displayed saliently with great emphasis. Be conservative if character clearly- Better to safe than sorry. 3. Fornat is important. Please keep the stringe identical formattings
cannot identify
post post-proceesiag.
Please make the [1. Appeared Characters] and [2. Character Datails] consistent.
Now your job hegins.
Table D: "Character Selection" prompts at the Captioning stage (Sec. D.1).
26

---

# 1. Inatructiona You given fev continuoue screeushota objects of the video in your nemory, and the caption of the previous scene. Your objective is to generate caption for current displayed scene. Your analysis will video.
clip, sone potential key characters
accurate to the
Inpat:
representative video franes, and the identifiable propartiea the characters and objects in the visual nemory.
Visual Nanory: the
caption 3. Video Franes: the fex contimuous screenshots of the current video clip.
Previous Caption:
previous scene.
# Guidelines
Whon generating the caption, please follos the guidelines belox and solve this problen step by step:
First, describe the content the
briefly.
2. Second, use the visual nemory to ideatify if any characters or objects fron the vieual nenory scene.
so, please 3. Third, describe the scene in detail, including the characters, their actions, the objects, the properties the charactera and objects,
appear in
their out.
enviromment, and other typea contents, etc.
Sone detailed tips:
When generating the captions, please take the previous scene as conterts and protand that you goal
vatching video continuously.
human should your captions
watching continuous video.
When generating the captions, please faithful to the acenes.
logical
betveen
When you encounter characters, please utilize the information name from the visual nenory if What you see nstchee the vieual nemory. For instance, if the visual nemory contains character naned "arrsona"", you should use <parson_a> to refer to the character in your captions.
Inportant
1. The quality of this step is very very very inportant.
very detailed and faithful to the follosing aspacta:
vant you
least, you should go over
2.1 What are the charactera, what are their appearances, what are there clothea, what are their actiona,
what their emotione?
2.2 What are the objects, shat are their propertiea, what are their relationshipa with the charactera? are environeents, what the background, day?
the vesther, what
2.4 Are there any on
thay?
2.5 If there is anything salient or anything weird, please describe it.
# Fornat
Your reaponse should be in the following format:
[1. Brief Description]: ... captions, string
[2. Appeared Characters]:
the fornat of [NAMEL, HAME2, ...], list of character or object
names
[3. Detailed Deacription]:
the detailed description the acene, string
# 4. Your Job
Now your job begins.
Table E: "Dense Captioning" prompts at the Captioning stage (Sec. D.1).
Map: Goal-aware Scene-centric Analysis. Based on the information required to answer the question, MR. Video first proposes customized queries for each question as in Table J and applies these queries to the VLMs.
Reduce: Answer Generation. The final step is combine the results of goal-aware scene-centric analysis with the global intention analysis to generate a final response. The prompts are in Table K.
D.4 Datasets
LVBench Videos. We clarify the unavailable videos from IVBench, mentioned in Sec. 4.1. LVBench requires users to download from YouTube with provided links to protect the copyright. As of March 1st, 2025, 4 videos no longer available on YouTube, IDs are: 28CIeC8cZks, idZkam9zqAs, WgV99Dv, gXnhqFOTqqI. After filtering them out, we have 1,492 out of 1,543 questions. Therefore, MR. Video still outperform the other methods by more than 5% even under the extreme assumption of counting the unavailable questions as "wrong" answers.
cannot evaluate them. Their
LVBennh Ablation Subset. We select the first video of each category from LVBench (cartoon, live, self-media, documentary, TV, and sports) and form subset for the ablation study, as mentioned in Sec. 4.1. Th six selected videos are: Cm73ma6lbes, TeQBTesZUOO (-RiD12RWQS, hROKtPgktO8, PSE2YPuv and CgvlgGxZRIE. They consist of 98 questions in total.
27

---

Instructiona You will be given two sets of frames captured from a video, describing several charactere or objects video. objactive is find any character abject appears both sets. so, please help ne locate the character object and find the better Trame representing the characters and objecte.
Input:
1. Set 1: the names, representative video frames, and the identifiable proparties of the characters and objects.
2. Set 2: the names, ropresentative video franes, and the identifiable properties of the characters objacts.
N 2. Guidelines and Tipe
This
advanced reasoning ability and check every iten and step carefully. The follosing guidalines are vary inpertant for you to finish this task:
easy task, please make
the first with the sacond set; (b) if so, find the better frame representing the character objact.
Please
problen
stepa:
if any
repeated
Please rely on both the video frane information and the identifiable propertiea to carefully
understand the charectere and objects.
3, When you are selecting the better frame for an object, please consider the folloving factors: (a) salient frame shoving the front face the character; (b) the frama should be the nost representative frane showing the character or object. charactere objects captured from different angles distances, please your best Judgement check if they are the same character objact.
the frane should be the
N Dutput Fornat
Please strictly folloz the format belov to ensure smooth prote-prccssing [Rapeated Characters and Objecta]: (Charactar_nae) Sataattr_pamelii__eett. Character_ ctttteer_aeeeenn______ Better,character www.cccaarrrreeeeeee namel), (Character_nane) iatiet__nnne2__.Bee___ maracter_name22 ianne__in___etttt_e___eii Set_et, Better_charactee wwt.ccccaarrrrreeeee _amee2)
...
The ansver lista all the repeated characters and objects in the two seta of framea, each tuple contains three itens describing the repeated character or object:
the of the character 2. Character #baracter_nam_in__ee waanaaan_aat_____ the nane of the character or object in the second set of frames.
Character jinnicrr_aa_____St____
the first of franes.
Better, wwwrcccaarraaaneee
character
or repeated character or object, nust be consistent with the nane in srrracter_aaeein__ee
Character paaee. in_Set_2.
An example output should be:
(Repested Charactere and Objects]: (person_a, person_b, person_a), (dog_a, dog_b, dog_b)
# 4. Your Jah
you receive
and their descriptions.
your responsea
with the information provided.
Table F: "Character Merging" prompts at the Captioning stage (See. D.1).
Instructions
You will be given description of a video clip, which potentially contsins sone charactere. After sone analysis, I decided change me modify the descriptions to the nav names.
characters objecte, your job help
Input:
old description of the video clip, containing the fields of Bricf Description, Appeared Characters, and Detailed Description. Modified List: list characters modified the OLD_NAME NEM
Old Description:
Output:
modified description and contenta, only with names changed.
output
clip strictly folloring original
# Guidelines
1. Dnly change the nanes, do not change the fornat or any contenta. update Brief Description, Appeared Charactere, Detailed Description. 3. Keep the names consietent.
characters
Detailed Description <HAM>>, please
Tormat.
Your
Now your job begins.
Table G: "Caption Modification" prompts the Captioning stage (Sec. D.
28

---

Motivation
You will conduct the firat step of long video understanding: ssperceiving short video segnenta** and user's questiones, By using ehort-segnent analyeis, you limitation of the nodel's context length for long videos.
seanalyze their
You
access the following information for current video segnent:
The frames sampled from the rideo, nach corresponding to scone in the captions. video captioning model, decomposed into
The captions of the video generated representing different video actions. Notably, ke have marked the potentially key characters or Boxever,
objects using the inconsistent tracking across frames), please use it with reasoning.
entirely reliable (e.g., missing charactere
# Output Formate
Please strictly following output
inportant post-prosessing:
[1. Reasoning];
(Your ressoning procees. Please be precise, concise, and clear. Meationing
evidence any.)
[2. Relevant Segments]: [(t_start, t_end), ...]... ..... (List the time range of the video segnents that information fron the captiona. if you think continuous period necessary for the question, marge then into single segnent. Return empty Hit if nome of the segnents are
are relavant to question. Please strictly follow
[3. Confidence Level]: ... (Your relevant.)
level.)
[4. Key Characters]: [(character symnоnyn in questica, identifiable properties or HAKE in captions), the question and identify Keep the list empty if the question is not related to any characters.)
....... (The key characters
neati
# 3. Instructions and Guidelines
*a Information Reliability
question (including the optione, if any)*+ to analyze the user's intention. The reliability of the information
principle
+ecombine
captione, video
is:
The question: raised the
the moat inportant and reliable. Video frames: reliable, but only covers a small porticn the video.
5. Captions: less reliable, but covering details, especially the "NAME" representing
character/abject nanes. You should conbine the information from the question and video franes when using captious.
Analysis Tips
Think carefully about how short video aegnent could contribute to long video understanding paying attention to the question and video segment contente. Soue exanples are: For question visual details, you should check the video segment **contains the that the user vantgee.
question
over period
as order
actione, you
should reason sechether this segnent can contribute part of the analysiste.
reason implication video segnent sccontains the key informationss that help you understand the story/actions.
question
story/actions
video, you should chack
Finding the key video segment is critical. If the user nentions clear criteria, such as specific character object, try
it repreciselyev and #trigorouslyes in your analysis. should potentially localize its captions, or clearly specify appearance properties. 4. Pay attention to the information reliability mentioned above.
8. If the queetion ssss for certain characters in the plot/story,
Inagine yourself watching video using the sampled frames the captiona.
smalyeis, pleaee provide the ressening process and confidence level betseen (almost guessing, no clear evidence being relevant to the question) to (almost certain, clear evidence being relevant the question).
When discussing
7. If the question should be ansvered with contexts, for "Relevant Segnents", you should include example,
most possible scene increase robustnezs. most posaible segnent is (10, 20), and its previous and next scenes are (5, 10) and (20, 25), then should make it (5, 25) so that the contents between two scenes won't be nissed.
scene
Your Input
The question: question caning with options.
2. The franes: 1ist of franea sampled from the video.
captions decomposed Each caption is the format of "(t_atart, t_end): caption". Time represented in seconds.
captions:
scenes representing video actions.
# 4. Your Job Starts
Table H: "Segment Intention Analysis" prompts at the Question Intention Analysis stage (Sec. D.2).
29

---

# 1. Motivation
You will conduct question asking about. The questiona fron the users night be vague or not self-contained. You sill complete the question by finding relevant video seguents, charactere/objedt how short video segnents contribute to the long video understanding. You will access the folloring information: A question.
intention analysis step of long video understanding: what the
Your analysis of short video segments: *iis the video segnent relevant to the question?*x
important information in this step. You will go through the analysia each seguent containing the following parts:
Your analyaia
Reasoning: ... (your explanation)
2. Relevant Segnents: [(t_start, t_end), ....... (The periods that are potentially relevant fron your analysia. Time represented saconda.)
Confidence Level:
(Your confidence level.)
Key Characters: [[character аyпnoпyn in question, identifiable proparties or NAME in captions), (The key charactera unreliable.)
nentioned in question
identify then. Could
# 2. Instructions and Guidelines Objectivea
Your goal to merge the information from at the video level. Your most critical autput for the downstrean parts are the "relevant segnenta" and "key characters", Notably, you carefully use your reasoning skills handle following iesues:
short videa segments into complete understanding
Segment-leve analysis night guess some relevant segnents characters for question. You should select the nost relevant segnents and charactera basad on video-level undarstanding, and ignore the less relevant ones.
Segnent-level analysis night contain contradicting infermation since they come from separate analysea. You should carefully nerge the information from different segrents, and provide reliable information
dovnstream anslyses stepe.
3. You should clarify hox the results from sognent-level can contribute to the long video understanding. example, se segnents.
"sun", "nerge", or "select" the information from individual
M Output Fornats
[1. Reasoning]:
(Your reasoning process. Please be precise, concise, and clear. Mentioning
evidence any.)
(2. Relevant Segments]: [(t_start, t_end), ...]... (List the time range of the video segnents that relavant to the question. Please strictly fellow the tine information
the analysis provided you. Merge the scenes if you think continuous period is necessary for the question.)
[s. Key Characters]: [[character symnonyn in question, identifiable properties or BAVE in captione),
....... (The key characters that nentioned in the question and hov identify then. Keep the list eupty if the question is not related to any charactere.)
question requires combining contexts from different seguente to ansver. If "yes", then this is a global question. If "no", then this is local question.)
(4. Local Global]: ... (Khether
vary important follow the format the relavant segnents section. Every segnent should ba foruat of (t,start, t_end), sspecially the brackets should be "()" and natcbed.
Principles and Tips
1. Think carefully sbout how a short video segnent could contribute to long video understanding by segnent contents. Sone examples are:
paying attention to question For question an visual details, you should check if the video segnent *scontaina the scane that the user vantase,
For question on information over period of time, such the order or the number of actions, you should reasen seshether this seguent can contribute part of the analysiete.
implication the story/actions in the video, you should check if the video segnent **containe the key information* that can help you underetand the story/actions.
For question the
Finding key video segnent critical. character of object, try use it "preciselyxx and serigroossyy* in your analysia.
nentions clear criteria, such specific
If the question aeks for certain characters in the plot/story, you should potentially lecalize its <MAE>> in the captions, clearly specify its sppearance properties.
4. Imagine yourself satching a video using the sampled analysis. Figuring out the flov of the plots
critical.
5. If the question is not really about the ssubole videons, do not apecify more than 10 relevant
segnents.
6. You should propose *eat least relevant segnentee. If you don't think any segnent is relevant, return noet likely segnent and say 'I have low confidence on the relevance of the seguents".
# 3. Your Jab Starts
Table I: "Global Intention Analysis" prompts at the Question Intention Analysis stage (Sec. D.2).
30

---

Notivation
longer period In this atep of long video understanding, you are naking praparations for calling vision-language frames. Specifically. you be given user's question video-level analysis from yourself. Based on such information, you will sspropose question to prompt the vialon-languaee models*< to analyze the video franes. You will access the following information:
models to analyze sampled
1. A question.
A video-leval analysis from yourself. It contains the follozing information:
1. Reasoning:
(Your explanation about which parts of the video are relevant to the queation.)
Relevant Segments: [(t_start, t_end), ....... (The pariods your analysis. Tine is represented in saconda.)
are potentially relevant
Key Characters: [ccharacter sуыпопуи in question, identifiable propertiee or XAME in captions), ...... (The key characters that nentioned in the question and how identify then. Keep the list empty if the question is not related to any characters.)
from different segnents anaver. If "yes", then this is global queation. If "bo", then this is local question.)
Local Global:
(Whather the quastion requires conbining
# 2. Inatructiona and Guidelines
Objectives
When thinking about the questions to ask, please pay attention to hov the step will sample the video frames for your questions. In practice, we will use tuo ways:
each relevant segnent, e.g., franes. In this vay, vision-language models can use your question to check the details of each seguent.
Sample video frames
for each segnent, saquentially. In vay, the vision- language models use your question check the flow of the plots conduct reasoning time.
Global: Sample vidao
Therefore, you should propose tuo questions:
1. A local question: what kind of detailed information or evidence should the vision-language nodels rind each segrent?
2. A global question: what kind of ressoning should the vision-language nodels conduct on a longer
time span?
sa Output Formate
Please strictly follow the output formats belov to propose your questions, so that the dovnstrean parts the information:
casily
(Your reasoning process. Please be preciae, concise, and clear. Explicitly thinking about kind information nissing inportant for question.)
[t. Reasoning]:
(2. Local Question]:
(Your question for the local analyeis.) [3. Global Question]: ... (Your question
global analysis.)
Principles and Tips
Think carefully about how a short video segment could contribute to long video understanding by Sone examples For question on viaual details, you should check if the video seguent **containa the scene that the user vantg*e,
paying attention to the quastion and video sagnent
For question on information over peried of time, such as the order or the number of actions, you analysiste.
segnent contribute part implication of the story/actions in the video, you should check if the video segment *econtinns the kay informationee that can help you understand the story/actiens. clear, and within depending on any tine information.
should reasen
For question the
Keep your quastion
explicitly
Remember
the options
original questions, expressed
think about the best way to distinguich the correct one. It is also important to include the original options as
vasion-language Use your knovledge of prompting large language models vision vision-language langunge models improve your quastion.
output questions abould only contain question options. speculatione, or reseoning into the queetion. For exanple, the question should directly start ss DDscriie ... (A) .... (B) .... .... .... (B) ...", (D) (E) or similar formats.
include any analysea,
... (A) ... ....
# Your Job Starts
Table "Customized Queries for Perception" prompts at the Goal-aware Analysis stage (Sec. D.3).
31

---

# I. Motivation You are at of your analysie to finally ansver the user's question.
step long video understanding. You
user's question series
Before conducting actual analysis, is important understand the steps previoous analysis that will presented to you:
1. Video-level User Intention Analysis: You first analyze which parts of the video and what kind of characters relevant the user's question. You also think about how each video segnent could contribute to the long video understanding.
seguents, you proposed two questions for the VLMs to use. The first question is called "local question", used for detailed second question and reasoning acrose multiple zegnents.
Goal Propossl:
vieion-language nodele analyze
analysis for
segnent,
"global question", used for joint analysis
3, Goal-aware Analysis: You will receive the results the vision-language nodels' perception for video segnent using the lecal question and By understanding the previous stepe, you will have a good understanding of the neaning of the information provided you, especially which parts reliable and informative for ansvering the user's question.
nultiple segnents uaing the global question.
# 2. Instructions and Guidelines
Think carefully about how a shart video negnant could contribute to long video understanding by Sone examplea
paying attention to question and video segnent For question on visual detsils, you should check if the video seguent *scontains the scene that the user vants*.,
For question on information over period of time, such the order or the number of actions, should reason seshether this segnent can contribute part of the analysists.
For question on the reason implication of the story/actions in the video, you should check if the video segnent Mcontains the kay informationes that can halp you understand the story/actiana. analyais
Carefully consider whether important for ansvering the user's question.
segnenta or across multiple segnenta nore
With the information provided to you, imagine youself as the flow of the plots is critical.
vatching video. Piguring out
4. It is possible that sone information vague or contradicting each other. You should utilize edvanced ressoning skills resolve the contradictions. Sone very useful principles
If the user has mentioned a specific criteria, try to use it *sprecisalgas and *erigoroualyas in your analysis.
Try to utilize the confidence levels provided in the ansvers. Alxays thinking about your strategy:
segnents across multiple segnenta could contribute to the long vidao understanding. For example, do you combine the piecea information together, sunning some numbere, or picking the best segnent to auswer the question? Bunans have limited menory. Alvays prioritize the most salient information.
analysis
5. Pay attention to the tine information. They night provide additional correspondence information scross different segnente analyses.
# 3. Output
Please provide your anaver in the following fornat:
[1. Ressoning): ... (Your advanced reasoning based on the information above.)
please nake guess from à to E based on the information you have. To eneure correct poat-proceesing, please strictly use any characters spaces.) # 4. Your Job Starts
(2. Ansuer]: A capital letter from E (If you cannot find
this fornat.
Table K: "Answer Generation" prompts at the Goal-aware Analysis stage (Sec. D.3).
Breadth Benchmarks. As mentioned in Sec. 4.1, we utilize several long video benchmarks in addition to LVBench [39] to provide comprehensive evaluation. However, we evaluate on their subsets due to limited computation resources. (1) Long VideoBench [47]. We evaluate the official long video validation subset of Long VideoBench, containing videos with a duration (900, 3600] seconds. There are 188 videos and 564 questions in total. In the comparison. the accuracies of the VLMs come from Table of the Long VideoBeech paper. (2) EgoSchema [31]. We evaluate on the validation set of EgoSchema, which contains 500 vidcos and questions. The performance mainly comes from Table 2 of VCA [53]. (3) Video-MME [9]. Our evaluation follows the long video subset of Video-MME, under the setting of not using subtitles. This set contains 300 videos and 900 questions in total. The performance of models comes from the as March 1st 2025.
htttps:/vvieo- miceddo--meeeithuu...b.ee... wwt.hooeoaaaeeeeeeeeeeeeeeeeeeee hwtp:eppa.....alleeeeeeeeee htaddeboodrr
32

---

You are helpful assistant with the ability of vatching videos and anavering the questions raised process continuous screenshots raised by hunan useze. If you encounter any issues that you cannot ansver the question, please pick possible
by users.
video,
questiona
options.
Nhen you ansver, please follow the fornat of: [1. Reasoning):
(aay you choose this anewer) [2.
Ansver]: ... (The ansver you chease, from 4, B, C, the question, please pick the most possible from A, C, Do not leave it blank or select other optiona.
Inportant:
If you
Table The prompts used for evaluating Gemini-2.0 Seanini-2.0-Pae 2.0-Fllsh (Sec. D.5.1).
D.5
Analytical Experiment Details
D.5.1 Baseline Evaluation
As mentioned in Sec. 3.2, we evaluate Gemini-2.0-FFlash on the long video benchmarks. For the 30min hour ones, including LVBench, Long VideoBench, and Video-MME, follow the standard setting of uniformly sampling 256 frames from each video. For EgoSchema, whose videos are 3min, uniformly sample 128 frames for evaluation. With LVBench frequently asking about events of specific timestamps, we further provide each frame's seconds as interleaved images and texts. Since Long VideoBench's 's questions are commonly related to the subtitles, provide the subtitles of sampled frames as the contexts to the VLM. The prompts used for evaluation are Table I..
D.5.2 Ablation Study on Finding Relevant Segments
This section deseribes more details about the analytical experiments conducted in Sec. 4.4, where compare the question intention analysis of MR. Video with multi-modal retriever, MM-Embed [25].
Types of Questions. Since LVBench's s annotations for "summarization" and "reasoning" questions might specify long ranges, our evaluation mainly focuses on the question types with precise intervals: key information retrieval, event understanding, entity recognition, and temporal grounding. On our subset for analysis, this results in questions.
MM-Embed's Retrieval. Following the practice of VideoAgent [42], every video frame is encoded by concatenating image content with a timestamp since some questions are related to specific seconds. In addition, every question is encoded along with its multiple choices, as some questions do contain specific contexts. Since MR. Video might propose multiple candidate scenes, we let the retriever select the same number of top-k candidates for fair comparison. Finally, every question searches its relevant frames via the maximum inner product between the question and video frame embeddings.
E Broader Societal Impact
As general principle for long video understanding and its corresponding agentic framework, our MR. Video does not introduce societal bias in this process. However, the MR. Video framework utilizes the VLMs and LLMs, represented by G.mini-2.0-Flash and GPT4o, so the results might indicate the societal biases of these models.
33


Reviews:
[{"review_id": "rUuZ9VN0Y5", "reviewer": "Reviewer_Zpo3", "summary": "This paper introduces MR. Video, a novel long video understanding framework inspired by the MapReduce paradigm. The system processes video in two MapReduce stages: one for generating consistent, detailed captions and another for question-specific reasoning. It tackles the core challenge of jointly handling dense local perception and comprehensive global reasoning under limited context length constraints. Experiments on LVBench dataset show improvement over previous methods.", "questions": "- While the authors acknowledge using GPT-4o and Gemini-Flash for cost control, so how is the performance of proposed approach with fully open-source models (e.g., Qwen, InternVL) instead of proprietary models?\n\n- The main comparisons rely on different model configurations (e.g., MR. Video uses Gemini and GPT-4o, while other, like in VideoAgent, use GPT-4o alone). Could the authors clarify how they ensured fair comparisons? For example, are the gains solely attributable to better orchestration or also due to stronger base models?\n\n- The ablation study reports gains from consistent character naming and intention analysis. Could the authors elaborate further on how robust these components are across videos with high character diversity or minimal dialogue?", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "***Strengths***\n-\tThe reformulation of long video understanding as a MapReduce process is particularly interesting and original. \n-\tThe paper presents two concrete Map and Reduce stages with clear definitions for each step, making it generally easy to follow.\n\n***Weaknesses***\n- MR. Video uses proprietary models including GPT-4o and Gemini-Flash, which would limit reproducibility. Besides, I also concern about evaluation fairness against  other baselines using different LLM/VLM combinations.\n- There is no discussions of error propagation (e.g., how errors in captioning affect downstream analysis). This could raise a concern given the reliance on intermediate generated text.\n\n - The qualitative examples used in the paper appear to be animated or stylized content (e.g., cartoons or anime) rather than real-world video data. It raises concerns about the generalizability of MR. Video to other data domain like real, unconstrained, or egocentric video settings, where visual noise, occlusion, and ambiguity are more prominent.", "was_revised": true, "final_justification": "As noted in my comments on the rebuttal, I decided to maintain my positive rating on the contribution of the paper.", "rebuttal": "We are grateful for your positive feedback! We are encouraged that you recognize the simplicity and effectiveness of our MapReduce design for video agents. We appreciate your constructive questions and will clarify them below.\n\n## 1. Proprietary Models & Fair Comparison\n\n**Common Practice of Using Proprietary Models in Video Agents**. We appreciate your raising this question and recognizing our challenges in expense control. The main reason we use proprietary models is to follow the common practice in most existing video agents, such as VCA [1].\n\n[1] Yang et al. VCA: Video Curious Agent for Long Video Understanding. ICCV 2025.\n\n**Difference between Gemini-2.0-Flash and GPT4o.** Second, we clarify the difference between Gemini-2.0-Flash used in our paper and GPT4o used in video agents, *e.g.*, VCA, regarding fair comparison. GPT4o is commonly considered stronger than Gemini-Flash-2.0: In our Tables 1 and 2, GPT4o is better or has similar performance to Gemini-2.0-Flash. In addition, GPT4o's cost is 250x higher than Gemini-2.0-Flash (2.5 dollars v.s. 0.1 dollars per 1M input tokens), which also indicates that GPT4o is a generally stronger model. \n\n**Comparison: Other Video Agents in Gemini-2.0-Flash.** We further add the experiments of evaluating the previous video agents with Gemini-2.0-Flash instead of GPT4o as an example of fair comparison. We mainly include VideoAgent and VideoTree because they have open-sourced their code. We conduct this on LongVideoBench, identical to the experiments shown in Table 2. As shown in the table below, our MR. Video still outperforms previous video agents after switching their VLMs to Gemini-2.0-Flash for a fair comparison. (We omit running our MR. Video with GPT4o as GPT4o is stronger, and our limited expense and time during the rebuttal.)\n\n| Method               | Acc (GPT4o)             | Acc (Gemini-2.0-Flash) |\n|----------------------|------------------------|------------------------|\n| Direct VLM Inference | 58.6                   | 45.7                   |\n| VideoAgent           | 47.6                   | 37.8                   |\n| VideoTree            | 39.2                   | 42.6                   |\n| MR. Video (Ours)     | -     | **61.6**                   |\n\n**Open-source models.** Third, we add the suggested experiments of using open-source VLMs and LLMs: Qwen2.5-VL-72B as the VLM, and Qwen2.5-72B as the LLM. We follow the same setting of our ablation experiments (Sec. 4.4). An engineering issue we encountered was the high GPU demand and price of running the 72B model on numerous frames. So we utilize the APIs from OpenRouter, which runs stably with less than 32 frames. In the comparison, we run the baseline of Qwen2.5-VL-72B model for direct inference using 32 frames, and let our MR. Video to utilize $\\leq$8 frames to show the effectiveness of using shorter context lengths. Finally, our MR. Video outperforms the Qwen2.5-VL-72B baseline by 45.9% over 38.8%. This suggests that our framework is also applicable to open-source models.\n\nTherefore, our use of proprietary models is (1) consistent with the video agent's standard practice; (2) utilizing a weaker Gemini-2.0-Flash as the VLM; and (3) can also improve open-source Qwen models. \n\n## 2. Analysis of Captioning's Error Propagation\n\nThe error propagation from captions to the final results is an interesting and important topic. Unfortunately, this problem still does not have an established analytical framework, to the best of our knowledge. Therefore, we provide the following empirical observations:\n\n1. Quantitatively, our ablation study in Fig. 6 analyzes \"what if the captions do not model consistent characters?\" The drop in accuracy (63.3% to 58.2%) suggests the influence of caption quality on the downstream analysis: if the caption quality decreases, the final question-answering accuracy will be impacted.\n2. Qualitatively, we observe that the subsequent video agent components will fail if the captions overlook crucial information. In example 2 of Fig. B (supp), we discussed the captions overlooking the details of a key frame (noodles on the woman's face), which caused this frame to be ignored by the later stage of the MR. Video. \n3. Our design also implicitly alleviates the error propagation from captions via incorporating raw visual information: specifically, our segment intention analysis (Sec. 3.3.1) sparsely samples the middle frame of each scene (L192, main paper) to assist the localization of relevant scenes without entirely relying on the captions.\n\n## 3. Real-world Videos & Generalization\n\nThank you for looking at our examples closely!\n\n**Our MR. Video can generalize to a wide range of videos** (real and unconstrained video settings). Concretely, the major benchmark LVBench (Table 1) covers diverse videos beyond anime, such as live TV, documentaries, etc. Moreover, we also evaluate on a broad range of benchmarks (Table 2), including the datasets of LongVideoBench, VideoMME, and EgoSchema, where our MR. Video achieves a consistent advantage. These suggest the generalization of our framework.\n\nRegarding \"qualitative examples ... appear to be animated contents,\" perhaps your confusion is why the paper mainly uses the anime example in Sec. 3. This is, in fact, a deliberate choice to improve the readability of the paper. As mentioned in the footnote on page 3, \"The displayed video is the 1st from LVBench. We will consistently use it for method demonstrations for readers’ convenience.\" We thought that such consistency would make it easier for the reader to understand. Please let us know if you have further suggestions on this part.\n\n## 4. Robustness of Character Naming and Intention Analysis\n\n**Character Diversity.** We are one of the first video agents trying to use VLMs for human-like consistent naming, and we indeed notice that the reliability of consistent naming is limited by the capabilities of existing VLMs, *e.g.*, Gemini-Flash. Even so, we notice that:\n\n1. The VLMs are good at identifying celebrities or famous characters consistently, *e.g.*, the player Salah is recognized throughout the video in the example of Fig. 7 (left).\n2. For less well-known characters, VLMs primarily rely on facial features to conduct the matching. For example, we checked the example video from Fig. 7 (right), including the later video parts that are not presented in the paper due to space limits: the identification of Mikhail is consistent, despite his clothes being different in the later parts of the video.\n\nAlthough we have observed several successful cases, we acknowledge the inherent challenge of consistent naming. Specifically, we check the captions for the video shown in Fig. 2, where the main character has demonstrated multiple ages, clothing, and camera angles. Although our pipeline manages to merge some of them, it is hard for the VLM to realize that these are different stages of the same character. Therefore, we hope our study can initiate the investigation into long-term character consistency by illustrating its utility in video QA (Fig. 6), so that this problem can be better noticed and addressed in future works via better frameworks and VLMs. \n\n**Intention Analysis.** We assume the reviewer is asking about whether the intention analysis depends on dialogues. In fact, most of the benchmarks we evaluated, such as LVBench, do not use dialogue. Therefore, our MR. Video mainly uses the visual content and the logic of the video to analyze which segment is relevant to the questions in the intention analysis. From this perspective, our framework is robust to videos with minimal dialogue."}, {"review_id": "8d33jsnsiS", "reviewer": "Reviewer_FvH9", "summary": "MR. Video introduces a MapReduce-inspired framework to handle long video understanding. It breaks a video into short, densely captioned clips (\"Map\") and then jointly aggregates their information (\"Reduce\") for deeper reasoning. It explicitly makes characters consistent across different scenes for coherence. On the LVBench benchmark of hour-long videos, MR. Video boosts accuracy significantly, reaching 60.8% accuracy.", "questions": "1. Given the widely adopted thought that inference-time scaling improves accuracy, it would be better if the authors could provide the average number of tokens being used for all the methods. This is particularly crucial when comparing agent-based methods, as their system prompts may vary significantly. This may provide a more comprehensive comparison.\n\n2. It would be interesting to see how the reasoning model would work in this setup. I'm just curious about the performance. The result being better or worse will not affect my final scoring.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n1. The Map-Reduce principle is simple and general for long video understanding. \n\n2. The consistent character/object names in the captions are critical. It also benefits the reasoning of actions and stories across long horizons.\n\n3. Clear improvement over previous methods on LVBench.\n\n## Weaknesses\n\n1. The character name takes the template of “<entity>_<index>”. I am curious how the set of entities is built and which entity is chosen. Using Figure 3 as an example, my question would be why the character in Scene 1 Shot 1 has to be <samurai_a> rather than <people_a>?\n\n2. The paper has not explored to what extent character consistency persists. What if the character changes appearance (due to aging) or their attire (due to different plots)?.", "was_revised": true, "final_justification": "I appreciate the authors for providing the rebuttal. My concerns are addressed. I particularly like the analysis on \"3 Token counts\". I suggest adding the discussion in the final revision, in which case, you will have more time to analyze other baselines beyond VideoAgent.", "rebuttal": "Thank you for your positive feedback! We are glad to know that you find our MapReduce principle simple and general for long video understanding, our improvement significant, and that several other designs sound beneficial to you. Regarding your concerns and questions, we clarify as follows:\n\n## 1. Design of Character Names\n\nOur key intuition to achieving human-like character understanding is to recognize all the salient objects and characters that could catch a human’s attention. To achieve this, we design the following steps:\n\n> how the set of entities is built and which entity is chosen \n\nThe entities are built by instructing the VLMs with our prompts (Table C) designed to identify the salient characters and explicitly distinguishing them via a special naming convention. The three crucial steps are:\n\n1. Sparsely sample video frames (L155), specifically, 30 frames for each 3min, for the video to identify salient characters and objects; \n2. Instruct the VLMs to \"identify the potential key characters and objects from the video\" (Prompts in Table C); \n3. Provides in-context examples to explicitly ask VLMs to name the characters \"in general names, such as person_a, person_b\" (Prompts in Table C). \n\n> Using Figure 3 as an example, my question would be why the character in Scene 1 Shot 1 has to be [object Object] rather than [object Object]\n\nBased on the above discussion, we design the human-readable format of `[{object}_{index}]` for the VLM to follow, *i.e.*, the ability to generate names like < samurai\\_a> mainly comes from our in-context examples and VLM's predictions.\n\n## 2. Persistence of Character Names\n\nThis is indeed one of the most challenging problems in long video understanding, but still does not have an established solution or analytical framework, to the best of our knowledge. Therefore, we focus on providing an empirical discussion for your questions. We will also gladly include these discussions in the revisions of the paper.\n\nSince we are using VLMs to conduct consistent naming, the performance is primarily bottlenecked by the capabilities of existing VLMs, *e.g.*, Gemini-2.0-Flash, and will improve in the future. Even so, we notice that:\n\n1. The VLMs are good at identifying celebrities or famous characters consistently over a long sequence of time, *e.g.*, the player Salah is consistently recognized throughout the video in the example of Fig. 7 (left).\n2. For less well-known characters, VLMs primarily rely on facial features to conduct the matching. For example, we checked the example video from Fig. 7 (right), specifically the character of \"Mikhail.\" We found that the identification of Mikhail was consistent throughout the video, including the later video parts when his clothes were different. (The latter parts of this video were not displayed in the main paper due to space limits. To incorporate this discussion, we will include more frames of this video in our revisions.)\n\nDespite observing successful cases, we acknowledge that identifying consistent characters remains challenging in long videos for existing VLMs. Concretely, we check the captions for the video shown in Fig. 2, where the main character has demonstrated multiple ages (childhood and adult), clothing, and camera angles. Although our pipeline manages to merge some of them, VLM struggles to realize that some scenes are different stages of the same character. \n\nFrom this aspect, we hope our study can initiate the investigation into long-term character consistency by illustrating its utility in video QA (Fig. 6), so that this problem can be better noticed and addressed in future works. \n\n## 3. Token Counts\n\nAccording to your suggestion, we analyzed the average token count comparison. Because of limited resources and time during the rebuttal, we primarily compare our MR. Video and the representative open-source VideoAgent [1] on LongVideoBench, following the setup of Table 2. We found that your insightful question is a unique angle to demonstrate the distinctions and advantages of our MapReduce principle compared with previous video agents.\n\n[1] Wang et al. VideoAgent: Long-form Video Understanding with Large Language Model as Agent. ECCV 2024.\n\n**3.1 Token Count Comparison** \n\nBeginning from the same captions, our MR. Video consumes approximately 14x more tokens than VideoAgent to achieve a significantly better accuracy, as shown in the table below. Such a difference comes from the fact that:\n* VideoAgent only allows the agent to interact with the video for at most four rounds, while \n* our MR. Video requires the agent to densely perceive the whole video. \n\nThen a potential follow-up question arises: **Is such a design of using more tokens to achieve better accuracy valid?** \n\n|                  | Avg Input Tokens Per QA | Avg Output Tokens Per QA | Accuracy |\n| ---------------- | ----------------------- | ------------------------ | ---------------- |\n| VideoAgent       | 7695                    | 383                      | 47.6\\%             |\n| MR. Video (Ours) | 109522                   | 4908                     | 61.6\\%             |\n\nFor MR. Video, such behavior is deliberate to achieve more reliable video understanding, and this is also the major distinction between our MapReduce principle and the previous video agents: **the agent is necessary to utilize more tokens and conduct a comprehensive analysis of the video**. In the paragraphs below, we explain this comparison step by step via three questions.\n\n* Why and how MR. Video aims at utilizing more tokens for more comprehensive analysis?\n* Is it trivial to improve video understanding by simply using more tokens?\n* Why do we claim the MapReduce principle to support better inference-time scaling while it requires more tokens?\n\n\n**3.2 Necessity of Using More Tokens for Comprehensive Understanding**\n\nVideoAgent commonly relies on key frame retrieval, where it checks the video for an upper-bound of 4 rounds, while our MR. Video emphasizes intention analysis, where a VLM has to understand the whole video content comprehensively. This is the primary stage where our MR. Video utilizes more tokens. Such a design stems from our key insight: **the necessity of a comprehensive understanding of the video, without the reliance on key frame retrieval**. In the paper, we ground this proposition with the examples of counting numerous events (Fig. 1) and the common vagueness in long video questions (the identity of \"protagonists\" in Fig. 2), suggesting that video agents cannot make any assumptions about the questions and video contents. \n\n**3.3 Non-trivial Test-time Scaling**\n\nUnder the principle of comprehensive video understanding and aiming at the benefits of test-time scaling, it is non-trivial to directly improve accuracy by simply spending more tokens. For example, Fig. 3 (left) of VideoAgent suggests that **increasing the round of video perception would saturate or even decrease the accuracy**. Such an observation motivated us to explore a new paradigm of test-time scaling for long video understanding: instead of pursuing the direction of **depth** in previous video agents, *i.e.*, increasing the rounds of searching for key frames, our MapReduce principle introduces another direction of improving the **breadth** of understanding, *i.e.*, optimizing for the coverage of information. Driven by this new direction, our MR. Video converts the additional tokens into covering every part of the video and effectively leads to an improvement in accuracy.\n\n**3.4 MR. Video's Advantages in Inference Scalability** \n\nFinally, we clarify the seemingly contradictory point: how can MR. Video be more friendly for inference-time scaling while utilizing more tokens? The key insight is to consider the **critical path** of answering a user's question correctly when we serve a video agent framework on systems. \n\nTake the scenario in Fig. 1, for example, where both VideoAgent and our MR. Video need to count > 50 events of goals: (1) VideoAgent relies on iterative key frame retrieval, so its critical path is more than 50 steps and could be even longer for more complex videos or multi-hop reasoning; however, (2) our MR. Video mainly relies on parallel steps (“Map” in MapReduce) and shows a controllable critical path of 3 MapReduce stages. As a result, **our MapReduce principle is advantageous in using parallel computation to enable inference-time scalability**, *i.e.*, **the duration of videos that could be processed in unit time can scale linearly with the number of GPUs** (or VLM inference bandwidths) under our design.\n\n**To conclude**, we hope the above discussion both answers the reviewer's inquiry on token counts and clarifies how this marks the distinctions and advantages of our MapReduce principle for test-time scaling. We would be glad to discuss with the reviewer for any follow-up questions.\n\n## 4. Reasoning Models \n\nThanks for the suggestion! We shift our previous GPT4o to Gemini-2.5-Flash (with thinking ability) for handling text processing in MR. Video (note: our vision component still uses Gemini-2.0-Flash, which does not have thinking abilities). We experiment following our ablation study setting (Sec. 4.4), and the accuracy improves from 63.3% to 66.3%. Therefore, **reasoning model can benefit our MR. Video**."}, {"review_id": "4uyrIZVX6x", "reviewer": "Reviewer_b2Lt", "summary": "This paper introduces the MapReduce principle from big data processing to long video question answering. The authors build a pipeline including understanding, reasoning, and answering, and apply the principle to all three stages. In the first stage, the authors generate dense captions for videos and associate characters in multiple captions by using the same tokens. Then a VLM is prompted to analyze the relevance between questions and captions in the second stage. In the final stage, an LLM is instructed to gather additional information using the VLM and generate the final answers. The evaluation shows improvement over previous approaches and verifies the effectiveness of the three stages.", "questions": "1. The first step of the pipeline is segmenting videos. Does how to segment videos affect the following stages or the performance? How do the authors design segmentation? Does any important information break across segments?\n2. Removing context length limits may be a huge benefit of this work, but in the Reduce stages, the model also needs to gather information across multiple video segments. Are the Reduce operations also subject to context lengths?\n3. Could the authors provide some examples or even a quantitative study to show the advantages in Fig. 1(d)?  For example, the approach may need more computation but operates in parallel, so what is the overall cost?\n4. Ablating character name consistency reduces performance less than I expected. Could we analyze why the framework can answer a large proportion of questions despite inconsistent naming?\n\nOverall, I would be happy to raise the score if the questions can be properly addressed.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. The concept of MapReduce for processing long content is interesting and reasonable.\n2. The proposed approach shows significant improvement in multiple categories and datasets.\n\nWeaknesses\n1. This work designs a dedicated test-time scaling tailored to long video question answering. It provides insight on how to properly understand videos and answer questions step-by-step, but the technical novelty may be limited.\n2. The pipeline involves captioning and multistage analysis with a VLM and an LLM, which may require huge inference computation.\n3. The authors highlight the advantages of MapReduce against seq-to-seq VLMs and video agents. However, the advantages are not verified in the evaluation.\n4. (Minor) All figures contain a lot of small text, which might not be reader-friendly.", "was_revised": true, "final_justification": "The authors respond to my concerns and questions thoroughly. I raise my score and recommend acceptance.", "rebuttal": "We are thankful for your constructive feedback and suggestions! We truly appreciate your recognition of our MapReduce principle as \"interesting and reasonable\" and find our approach leading to \"significant improvement in multiple categories and datasets.\" For your questions and concerns, we have prepared the following discussions:\n\n## 1. Technical Novelty\n\n**Intellectual Contribution.** Our MR. Video **clarifies a critical myth in the previous video agents** and **conceptualizes the solution into a simple MapReduce principle**. \n\n1. In contrast to the reliance on key frame retrieval in previous video agents, *e.g.*, VCA [1], we explicitly emphasize **the necessity of comprehensively understanding video contents**. We ground this idea with the examples of counting numerous events (Fig. 1) and the typical vagueness in long video questions (the identity of \"protagonists\" in Fig. 2), suggesting that video agents cannot make any assumptions about the questions and video contents.\n2. Based on the above proposition of comprehensive understanding, we **conceptualize the MapReduce principle** to overcome the limitations of previous VLMs and video agents. (a) For VLMs, MapReduce is not limited by context lengths. (b) For video agents, MapReduce not only **improves video understanding via comprehensively covering all the segments** but also enables **better inference scalability from a system perspective**. Specifically, the critical path of MR. Video is shorter and more controllable than video agents relying on sequential steps. Considering the counting example in Fig. 1, previous video agents need >50 sequential steps to cover these events, while our MR. Video exhibits a controllable number of 3 MapReduce stages. (More explanations in the next section)\n\n**Method Contribution.** We also propose specific MapReduce steps with insights tailored to video understanding:\n\n1. *Consistent character names* in the captioning stage, which is essential for multi-hop reasoning (ablation in Fig. 6, qualitative example in Fig. 7, right).\n2. *Intention analysis* replaces key frame retrieval and more reliably identifies relevant video segments, as in the ablation study of Fig. 6, right.\n\n**Performance Improvement.** Finally, our proposed principles and designs demonstrate a **significant and non-trivial improvement** over the previous video agents, *e.g.*, >19% better than VCA [1] on LVBench. \n\nWe hope the above clarifies our novelties, and we are also glad to find that the other reviewers find the \"MapReduce process interesting and original\" (Zpo3), and \"MapReduce principle simple and general\" and \"consistent character/object names in the captions critical\" (FVH9). We would be glad to discuss further if you have follow-up questions.\n\n[1] Yang et al. VCA: Video Curious Agent for Long Video Understanding. ICCV 2025. \n\n## 2. Cost and Advantages of MapReduce\n\nFollowing the novelty discussion above, we clarify your two tightly connected questions.\n\n> The pipeline involves ... huge inference computation\n\n> … the advantages of MapReduce against seq-to-seq VLMs and video agents …\n\nFirst, we suggest that the evaluation (Tables 1 and 2) already highlights **the advantages of our MR. Video in terms of accuracy**. In addition to performance improvement, our MapReduce principle also demonstrates advantages from a system perspective, as demonstrated below.\n\n**Background: Necessity of Comprehensive Video Contexts**. Before the actual comparison, we would like to reiterate the key background (please kindly refer to the previous section for detailed discussion):  The necessity of comprehensive video understanding. Therefore, the comparison of computational costs is not meaningful if some methods deliberately digest incomplete contexts, *e.g.*, VLMs with sparse frames and video agents selecting a few retrieved frames. \n\n**Comparison with VLMs.** Bearing this background in mind, we assume MapReduce and VLMs are handling the videos at the same frame density to cover comprehensive information, *i.e.*, $N$ frames: \n\n1. Token count. VLM's token count is equal to the visual tokens on the $N$ frames, denoted as $N\\times K$. For our MR. Video, only the captioning step needs to watch the video frames densely, spending approximately $2 N\\times K$ tokens (two passes of segmenting the videos and generating captions), while the other multi-stage steps use texts or sparsely sampled frames, only take $0.32 N\\times K$ on average. Therefore, our token count can be treated as expanding the test-time computation for better comprehension.\n2. Computation cost. Considering a common transformer-based VLM, it requires $\\mathcal{O}(N^2)$ computation for $N$ frames, while our MapReduce splits this into $S$ parallel segments and decreases the computation to $\\mathcal{O}(N^2/S)$. As the number of stages in MR. Video is smaller than the segments $S$ of a long video, and the total computational cost of our MR. Video is smaller than that of a VLM.\n\n**Comparison with Video Agents.** Similarly, we analyze the cost of existing video agents for $N$ frames:\n\n1. Token count. Both our method and video agents rely on dense captioning, thus, they cost similar token counts as MR. Video to achieve the same quality of captions. \n2. Critical path and parallelization. Our MapReduce principle enables better inference-time scalability and shows advantages in shorter \"critical paths.\" Consider the counting task in Fig. 1, where a previous sequential video agent and our MR. Video both have to count > 50 key events: (a) Video agents relying on iterative key frame retrieval would take > 50 steps in their critical path, and the number of steps also varies with the complexity of videos and multi-hop reasoning, while (b) MR. Video relying on parallel map steps shows controllable critical paths of 3 MapReduce stages. As a result, **our MapReduce is advantageous in using parallel computation to enable inference-time scalability**, *i.e.*, **the duration of videos that could be processed in unit time can scale linearly with the number of GPUs** (or VLM inference bandwidths).\n\nTo conclude, (1) our MapReduce is better in accuracy, and (2) a detailed comparison of computational costs better justifies our design philosophy and scalability from a system perspective compared to both seq-to-seq VLMs and previous video agents.\n\n## 3. Segmenting Videos\n\nIn our captioning, the first step is to segment the video into atomic scenes and conduct captioning for each scene (L141-152). The following paragraphs clarify your questions on this. \n\n**1. Motivation.** Existing VLMs might overlook details when the video has multiple transitions. Therefore, we segment videos into atomic scenes (or the concept of \"shot\" in films) to overcome these limitations of existing VLMs. Such an operation also saves redundant captioning, *e.g.*, a static scene of over 1min when people are talking.\n\n**2. Implementation.** We prompt VLMs to (1) decompose each video segment (*e.g.*, 10s) into atomic scenes according to transitions, then (2) for the scenes at the boundaries of 10s video segments, merge them if they show coherent actions. The prompts are in Tables A and B (supp). \n\n**3. Information Break.** To avoid such breaks, we (1) let the VLMs generate a brief caption during scene segmentation; and (2) use these captions from previous scenes as the contexts for dense captioning (Table D).\n\n**4. Influence of Scenes.** We find that scene segmentation influences the captioning quality and affects downstream performance. We add this ablation by direct dense captioning on 10s segments and running MR. Video: The accuracy drops from 63.3% to 60.2%, indicating the value of better designs in the captioning.\n\n## 4. Context Limits for \"Reduce\"\n\nWe clarify that VLMs are limited by context lengths in videos because of the large token number per image, which is commonly >200 (*i.e.*, for Gemini, each frame takes 258 tokens). Then, even sparsely sampling 128 frames can cost 32k contexts. \n\nIn comparison, MR Video's reduce steps only operate on texts, which is more efficient. Concretely, \"Global Intention Analysis\" (Sec. 3.3.2) requires the longest contexts in our “reduce” steps by digesting all the segment-level analysis. Since an hour-long video contains approximately 30 segments (L189, main paper), this is equivalent to LLMs analyzing an essay with 30 paragraphs. Its average token count is 4.5k, while the maximum is 12k, both significantly smaller than the VLM's contexts mentioned above.  Therefore, our reduce step is less limited by context lengths.\n\n## 5. Consistent Names\n\n> … answer a large proportion of questions despite inconsistent naming?\n\nThis is a very insightful question, especially from a benchmark perspective: the majority of the questions from existing long video benchmarks can be answered or speculated without modeling the consistency of characters. Take the two questions in Fig. 2, for example:\n\n1. \"How many sticks does the protagonist put in the incense burner?\" Even though the model does not identify \"protagonist,\" it can still correctly answer the question by finding the only scene with \"incense burner.\" \n2. \"Why does the protagonist tie iron to her limbs?\" This question seems to require the identification of the protagonist, but the LLM can utilize its knowledge of “tying iron” to speculate that this action is related to fitness.\n\nDespite benchmark limitations, we believe that consistent characters are essential for human-like video understanding and have shown their effectiveness. Showcasing its importance requires better benchmarks in future work to avoid the shortcuts of VLMs. We hope our demonstration of the benefits in the ablation study (Fig. 6) and quantitative observation in Fig. 7 (right), where the identification of a specific character \"Mikhail\" benefits multi-hop reasoning, and initiates attention into this domain.\n\n## 6. Texts in Figures\n\nThanks for the feedback, and we will fix this in the revisions following your advice."}, {"review_id": "IGzP5UvM88", "reviewer": "Reviewer_wmFm", "summary": "In this paper, the authors propose Mr. Video, a new video agent to realize long video understanding. It contains two part: map and reduce, with different prompts and strategies.", "questions": "Please refer to Strengths And Weaknesses", "limitations": "Please refer to Strengths And Weaknesses", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength\n1. The paper is well-written\n2. The results of the proposed method seem good\n\n### Weakness\n1. My main concern is fairness. Study of agent strategy with API models has a serious problem: the model varies with time, and we do not know the exact details. Such a black-box manner hinders the disentanglement of the performance between the improvement of the API itself and the effectiveness of the proposed agent strategy. So I think a study based on open-source SOTA models is necessary, for example, Qwen-2.5-VL-72B.\n\n2. Following 1, the comparison with previous agents is also not fair. As mentioned in lines 302-307, the baselines are not re-evaluated with the same API. I understand the problem of evaluation cost. But I do not think it is a reason to conduct an unfair comparison.\n\n3. I think the Consistent Name is an interesting and important part for an advanced agent. But Sec 3.2.2 is too brief for me to understand the exact solution to realize it. After I read the prompt from the supp. It seems it needs to check pairs of frame subsets for all the combinations to realize identity consistency; it seems costly. So it raises my concern about the reasonableness of the proposed solution.\n\nBased on the above problem, despite the good benchmark results, I give a Borderline reject to the current paper. I will adjust it based on the comments from other reviewers and the rebuttal.", "was_revised": true, "final_justification": "The rebuttal addressed my concerns, so I raise my score to 4.", "rebuttal": "We appreciate your insightful feedback and your recognition of the improvement we have achieved. We hope the following clarifications and additional experiments could address your concerns.\n\n## 1. Proprietary Models for Video Agents\n\n> My main concern is fairness. Study of agent strategy with API models has a serious problem...\n\nWe appreciate you raising this question and recognize our difficulty in controlling the expenses. We will first provide the essential background and move on to your deeper questions step by step.\n\nFirst, we use proprietary models following the **standard practice of existing video agents**, such as VCA [1], which is also compared in our paper. \n\n[1] Yang et al. VCA: Video Curious Agent for Long Video Understanding. ICCV 2025.\n\nSecond, we clarify the confusion caused by the Gemini-2.0-Flash used in our paper and GPT4o used in previous works: **GPT4o is commonly considered as stronger than Gemini-Flash-2.0**. As in our Tables 1 and 2, GPT4o is better or similar to Gemini-2.0-Flash. In addition, GPT4o's cost is 250x higher than Gemini-2.0-Flash (2.5 dollars - 0.1 dollars per 1M input tokens), which also indicates that GPT4o is a generally stronger model. (More quantitative results are in the next section, according to your request.) Therefore,  our MR. Video is, in fact, based on a weaker VLM.\n\n## 2. Fair Comparison with Gemini-2.0-Flash\n\n> As mentioned in lines 302-307, the baselines are not re-evaluated with the same API.\n\nWe evaluated VideoTree and VideoAgent with GPT4o, following their original design choices in the paper, while holding the thought that “GPT4o is generally better than our Gemini-2.0-Flash” as mentioned above. To fulfill your request, we re-run both of them with Gemini-2.0-Flash on LongVideoBench (same setting as Table 2) for a fair comparison. From the results below, we can conclude that:\n\n* GPT4o is a stronger model in most cases than Gemini-2.0-Flash. \n* Our MR. Video has a significant advantage under this fair comparison.\n\n|                      | Acc (GPT4o) | Acc (Gemini-2.0-Flash) |\n| -------------------- | --------- | -------------------- |\n| Direct VLM Inference | 58.6      | 45.7                 |\n| VideoAgent           | 47.6      | 37.8                 |\n| VideoTree            | 39.2      | 41.6                 |\n| MR. Video (Ours)     |  -    | **61.6**                 |\n\n(We omit running our MR. Video with GPT4o as GPT4o is stronger than Gemini-2.0-Flash, and our limited expense during the rebuttal.)\n\n## 3. Open Source Models\n\n> So I think a study based on open-source SOTA models is necessary, for example, Qwen-2.5-VL-72B.\n\nThat’s a good suggestion. We add the experiments of using Qwen2.5-VL-72B as the VLM and Qwen2.5-72B as the LLM. Because no previous published video agents utilizing Qwen2.5-VL-72B on long videos, we mainly conduct an ablation experiment: *whether our agentic MR. Video can improve VLM*. \n\nIn the experiments, we encounter an engineering challenge that inferring a 72B model on numerous video frames requires extensive GPU resources and expenses, so we utilize the APIs from OpenRouter, which commonly run stably with 32 frames. Accordingly, we compare (a) Qwen2.5-VL-72B uniformly sampling 32 frames from long videos, and (b) our MR. Video utilizing $\\leq$8 frames per VLM inference, to show the relative effectiveness of our video agent framework. Finally, our MR. Video outperforms the VLM baseline by 45.9% over 38.8%. **This suggests that MR. Video is also applicable to open-source models.**\n\nTo summarize our clarifications and attempts from the above sections:\n\n1. We used proprietary models in consistency with the video agents' standard practice; \n2. Our Gemini-2.0-Flash is commonly considered as weaker than the GPT4o used in other video agents; \n3. When other agents use Gemini-2.0-Flash for a fair comparison, our MR. Video still demonstrates significant improvement; \n4. The effectiveness of video agent frameworks can also be validated on the open-source Qwen model.\n\n## 4. Cost of Consistent Naming\n\nWe are encouraged that you also agree with the importance and interest of consistent naming, and appreciate the time you've spent with our detailed designs and prompts. Regarding your concerns about the cost of merging the character names across the video, let’s first recap the process of consistent naming and then analyze its computational cost in detail.\n\nOur consistent naming involves two steps:\n\n1. We first let VLM propose salient characters and objects using sparse frames on long video segments (3min). This leads to approximately 20 sets of characters for a one-hour video. The prompt for this part is in Table C (supp).\n2. Then we instruct the VLMs to merge the characters from different sets using the prompts in Table E (supp), which is the step you have efficiency concerns about.\n\nIn character merging, our process is **different from** checking all the frame pairs in $\\mathcal{O}(N^2)$, assuming $N$ character sets. Instead, we implement it to recursively **merge the adjacent two frame sets into one** until we have a single set of characters, which is $\\mathcal{O}(N)$ VLM calls in total, and most of the steps can be run in parallel. With the total number of character sets being approximately 20 for an hour-long video, this cost is reasonable. A side note is that all the video agents generally consider captioning as pre-processing and can be reused on the same video for answering numerous questions, which further decreases the cost concern in this captioning step.\n\nAs one of the first attempts at addressing name consistency in long video understanding, we hope the intuition above is reasonable for you. Our main design principle is to strike a balance between the two extremes of relying entirely on \"detecting and tracking objects exhaustively,\" which causes significant computational burdens, and \"only considering a few primary characters,\" which loses crucial information. We would be glad to learn if the reviewer has other feedback or suggestions."}]

Author rebuttal:

Meta-review: All reviewers found the adaptation of the map-reduce formulation to video understanding to be interesting, novel, and well-presented, while acknowledging the strong empirical results demonstrated across a carefully selected collection of appropriate datasets. Although the original reviews raised several concerns regarding the use of proprietary closed models, consistent character naming conventions, token count considerations, and evaluation fairness, the authors' rebuttals successfully addressed these issues, with all reviewers expressing satisfaction with the responses provided. The area chairs agree that this submission represents a valuable and compelling contribution to the field of long-form video understanding, offering both methodological innovation and solid experimental validation.
Decision: Accept (poster)