VideoLucy: Deep Memory Backtracking for Long Video Understanding
Authors: Jialong Zuo
Yongtai Deng
Lingdong Kong
Jingkang Yang
Rui Jin
Yiwei Zhang
Nong Sang
Liang Pan
Ziwei Liu
Changxin Gao
Venue: NeurIPS 2025 poster ()
Keywords: Long Video Understanding
Agent-based System
Primary area: applications

Abstract: Recent studies have shown that agent-based systems leveraging large language models (LLMs) for key information retrieval and integration have emerged as a promising approach for long video understanding. However, these systems face two major challenges. First, they typically perform modeling and reasoning on individual frames, struggling to capture the temporal context of consecutive frames. Second, to reduce the cost of dense frame-level captioning, they adopt sparse frame sampling, which risks discarding crucial information. To overcome these limitations, we propose VideoLucy, a deep memory backtracking framework for long video understanding. Inspired by the human recollection process from coarse to fine, VideoLucy employs a hierarchical memory structure with progressive granularity. This structure explicitly defines the detail level and temporal scope of memory at different hierarchical depths. Through an agent-based iterative backtracking mechanism, VideoLucy systematically mines video-wide, question-relevant deep memories until sufficient information is gathered to provide a confident answer. This design enables effective temporal understanding of consecutive frames while preserving critical details. In addition, we introduce EgoMem, a new benchmark for long video understanding. EgoMem is designed to comprehensively evaluate a model's ability to understand complex events that unfold over time and capture fine-grained details in extremely long videos. Extensive experiments demonstrate the superiority of VideoLucy. Built on open-source models, VideoLucy significantly outperforms state-of-the-art methods on multiple long video understanding benchmarks, achieving performance even surpassing the latest proprietary models such as GPT-4o. Our code and dataset will be made publicly available.

VideoLucy: Deep Memory Backtracking for Long Video Understanding
Jialong Zuo
Yongtai Deng'
Lingdong Kong² Jingkang Yang"
Rui Jin
Yiwei Zhang'
Nong Sang*
Liang Pan Ziwei Liu Changxin Gao
National Key Laboratory of Multispectral Information Intelligent Processing Technology.
School Artificial Intelligence and Automation, Huazhong University of Science and Technology NUS $S-Lab, NTU *Shanghai AI Lab {jlongzuo, egao)-hhut. edu. cn
Code & Dataset: videolucy github.
Abstract
Recent studies have shown that agent-hased systems leveraging large language models (LLMs) for key information retrieval and integration have emerged as promising approach for long video understanding. However, these systems face two major challenges. First, they typically perform modeling and reasoning on individual frames, struggling to capture the temporal context of consecutive frames. Second, to reduce the of dense frame-level captioning. they adopt sparse frame sampling, which risks discarding crucial information. To overcome these limita- tions, we propose VideoLucy, a deep memory backtracking framework for long video understanding. Inspired by the human recollection process from coarse to fine, Videol employs hierarchical memory structure with progressive granularity. This structure explicitly defines the detail level and temporal scope of memory different hierarchical depths. Through an agent-basel based iterative backtracking mechanism, VideoLucy systematically mines video-wide, question-relevant deep memories until sufficient information gathered to provide a confident answer. This design enables effective temporal understanding of consecutive frames while preserving critical details. In addition, we introduce EgoMem, a new benchmark for long video understanding, EgoMem is designed to comprehensively evaluate a model's 's ability to understand complex events that unfold over time and capture fine- fine-grained details in extremely long vidcos. Extensive experiments demonstrate the superiority of VideoLucy. Built on open-source models, Videol signifi- cantly outperforms state-of-the-art methods multiple long video understanding benchmarks, achieving performance even surpassing the latest proprietary models such as GPT-4o. Our code and dataset will be made publicly available.
Introduction
"Mom, I can feel my brain, the deepest parts of my memory. I can remember the feeling of your hand my forehead when I ran a fever. I remember stroking the cat, was so soft, a Siamese with blue eyes and a broken tail."
Lucy
Long video understanding [67, 44, 11] is highly concerned task, with the core objective of accurately and objectively answering various user questions based on entire video content. This process demands that system possess a comprehensive memory and grasp of almost all details within the video; otherwise, information gaps could lead to inaccurate answers.
"Corresponding Authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Diggaal
مии DO 000000
Oridial Video
0 مиم0и000
Hicrarchical Memory Structure
Sparse Frame Sampling
Dlaard taamall
Hierarchy- awaaee Frame Sampling
navall Captioning
Search Missing Information
Multi-level Captioning
At Stanere with Buee eyen and broken tax.
LOOP
O: What does the the call look Ml?
What does me enti loee Mar?
Mult Stage gent
A: There piot reiotee cats sideo
Mulli Agett
Memory Backtracking
Intrrationn
Collaboniton
Malli-leval Represstiatio
Coverage
Mulli-leva Bepresstation
2 Infermation Coverage
A Represmmttivve Existing Video A gent based Systems
Viduoluur (Dum))
Figure I: Comparison between our Videol with existing video agent-based systems [32, 47]. In (a). they usually perform frame-level captioning on sparsely sampled frames, and then search for information, resulting in great information loss and hampering temporal understanding. In (b), our VideoLucy, through hierarchical memory structure and a memory backtracking mechanism, effec- tively performs multi-level video representation and achieves comprehensive information coverage.
This need for comprehensive memory brings to mind a scene from the movie Lucy. The protagonist, Lucy, gains full access to her brain's 's potential due to accident, acquiring an exceptionally powerful memory. She can recall every detail of her life from birth, even the sensation of her mother stroking her forchead during infancy. This extraordinary ability to trace back and precisely capture all information, whether instantaneous frames or continuous events, is exactly the goal we pursue in the long video understanding task.
Recently, the agent- based systems [9, 48, 32, 59] have emerged as a promising approach for long video understanding. In contrast to traditional video multi-modal large language models (MLLMs) [58, 37, 63], which have difficulty handling extremely long visual inputs, the agent-based systems typically harness the reasoning, planning, and memory capacities of large language models (LLMs) [i]. They iteratively search for and consolidate key information related to the question, thereby facilitating a more proficient understanding of long videos. Nevertheless, owing to the ineffective modeling of comprchensive memory, these systems still confront two significant challenges.
On the one hand, they typically perform modeling and reasoning on individual frames, struggling to capture the temporal context of consecutive frames. Essentially, they utilize a pre-trained caption model [64] to generate text descriptions for each specified frame in the video. Then, taking LLMs [1, 15] as the core, they construct an iterative information search loop to obtain the question-relevan key frames and their supplementary descriptions. For example, DrVideo [32] retrieves set of key frames through the initial video frame-level documents. Then, it utilizes a multi-stage agent interaction loop to gradually update the text information of this set. However, in real-world applications of long vidco understanding, many user questions are closely linked to the temporal context of consecutive frames. These systems, which rely the information of individual frames in isolation, have relatively weak capabilities in temporal understanding.
On the other hand, they usually adopt sparse frame sampling to reduce the cost of dense frame-level captioning, which risks discarding crucial information. Obviously, generating captions for each frame of a long video demands vast computational resources and much time. For instance, even if we generate captions for each frame of one-hour video (with FPS), a total of 3,600 captions need to be produced and processed subsequently. Therefore, these systems adopt compromising measure, namely sparse frame sampling. For examplc, VideoTree [48] preprocesses videos by sampling the original frames at 125 FPS for Video-MME [11]. Explicitly, this sparse sampling compromise will lead to the loss of much crucial detailed information for long video understanding.
To address these limitations above, as shown in Fig. 1, we propose VideoLucy, deep memory backtracking framework for long video understanding. Drawing from cognitive science, human recollection typically proceeds from coarse to fine, starting with general impressions and gradually retrieving finer details. VideoLucy also employs a hierurchical memory structure with progressive granularity. This structure explicitly defines the detail level and temporal scope of memory at different hierarchical depths. As the memory delves deeper, the temporal scope shrinks dynamically while the detail level increases gradually. Additionally, with an agent- based iterative backtracking

---

mechanism proposed. VideoLucy systematically mines video- wide, question-relevane deep memories until sufficient information is gathered provide confident answer. In essence, just as humans move from fuzzy to a sharp memory when recalling the past, Videol also starts with a vague memory of the entire video and gradually delves into the detailed memories relevant to the questions. This design enables VudeoLucy to recall comprehensive memory of the entire video, achieving effective temporal understanding while retaining crucial information. Just like Lucy in the movie, VideoLucy can also say, "I can feel my brain, the deepest parts of my memory."
In addition, we introduce EgoMem, a new benchmark for long video understanding. Built on EgfLife [51], EgoMem comprehensively assesses a model's 's temporal understanding and fine-grained detail perception in extremely long videos. Through six creestion-asweering multi-dimensional evaluations on the model's comprehension of complex, time-evolving events in first-person daily life recordings. Additionally, EgoMem gauges the model's capacity to detect fleeting detailed visual features in long videos, such as those that manifest for merely a few seconds. contains 42 videos and 504 QAs in total, with each video averaging about 6.33 hours length.
designs, it conducts
We conduct extensive experiments to verify the superiority of our VideoLucy. Built on open-source models [2. 15], VideoLucy significantly outperforms state-of-the-arr methods on multiple long video understanding benchmarks [67, 11, 44], achieving performance even surpassing the latest proprietary models such as GPT-4o [1]. For example, on LVBench [44], our VideoLucy with Qwen2.5-VL-7B [2] as the captioner achieves an accuracy of 58.8%, representing an 9.9% improvement compared to GPT-4o. Ablation studies and analyses further validate the effectiveness of the hierarchical memory structure, and the "needle in a video haystack" experiment showcases its unprecedented detail perception capability. As an agent system with dynamic and comprehensive memory, our Videolucy can pave the way for future research in this rapidly evolving field of long video understanding.
2 Method
We present VideoLucyy, a deep memory backtracking framework for long video understanding, which dynamically recalls comprehensive and in-depth memories of the entire video based on the question, so as to achieve accurate answers. First, we propose hierarchical memory structure that conforms to the pattern of transitioning from coarse to fine in the human recollection process, thus enabling the efficient modeling of abundant information in long videos. (Sec. 2.1). Next, we present the agents which are empowered with different roles through prompt engineering, and these agents accomplish their respective tasks during the memory backtracking. (Sec. 2.2). Finally, we propose an iterative backtracking mechanism. Through multi- stage iterative loop pipeline, it dynamically explores in-depth memories related to the question, thus efficiently collecting sufficient information both in breadth and depth. (Sec. 2.3). In brief, based on having a vague understanding of the entire long video, Videol is designed to achieve accurate question answering by continuously expanding and delving deeper into its memory of the video, just like how humans recall past events.
2.1 Hierarchical Memory Structure
We demonstrate that a good memory structure capable of effectively representing abundant informa- tion of the entire long video should have the following characteristics.
1) Multi-level representation. Considering that the time ranges corresponding to the practical questions usually have a large span. For example, some questions focus the single-frame understanding of an instantaneous moment, while others require a holistic understanding of the visual content within a large time range. Then, an good memory structure should conduct modeling for this multi-grained temporal scope, in other words, it should be equipped with the capacity for multi-level representation.
2) Comprehensive Information Coverage. One of the prominent characteristics of long videos is the presence of vast amount of visual information. However, existing methods based on sparse frame sampling inherently result in significant information loss. Considering the uncertainty of the information quantity requirements of questions in practical applications, a good memory structure should be able to achieve comprehensive coverage of the information of the entire video.
Considering the above demonstration, we propose a new hierarchical memory structure, which, while effectively representing the visual content within multi-level temporal scopes, achieves comprehensive modeling coverage of the overall video information.
3

---

Ulb Alno Memory
Specifically, for video V with N frames, we use f to represent each frame, where the index of the frame in the video, and y/ is the label indicating which clip the frame belongs to. Then we divide the video into K short clips, and we use fy to represent each short clip, where k is the index of the short clip. The meaning of fi E2 if y/- k is that if the clip label y, of fi is k, then this frame belongs to the k-tth clip. Then for single clip Uk, the memory of it is formulated as:
Fine Meroryy
m& = VidCap(Vk. pk).
(1)
Course Memmory
where VidCap represents any video MLLMs [2, 69] that are capable of providing holistic text de- scription of a given video based on the instruction prompt Pk. We can observe that this formula essen- tially determines the temporal scope of the memory by explicitly constraining the number of frames in each clip. When K I, this memory degenerates into a holistic overview of the entire video. When K N, these memories represent detailed descrip- tions of each frame of the video.
Figure 2: A toy example of hierarchical mem- ory structure: For video clip of certain time segment, as the memory hierarchy deep- ens, the number of frames captured per sec- ond increases, and the time span covered by the memory shortens, thereby achieving multi- level video representations with comprehensive information coverage.
Then, we can obtain memories with different tempo- ral perception scopes by specifying different values for K. We explicitly define three types of memories with successively decreasing temporal perception scopes to form hierarchical memory structure. A toy example is shown in Fig. 2. From the shallow layer to the deep layer, they are respectively the long-range coarse memory, the short-range fine memory, and the frame-level ultra-fine memory. As clip division becomes denser, the number of text descriptions in these memories for the same-length video increases, achieving progressive memory detail granularity. With such a memory structure, we have not only achieved multi-level representation of long videos but also realized comprehensive information coverage.
2.2 Agents with Empowered Roles
As common practice for an agent-based system [39, 66), an MILM and LLM are empowered with respective roles to form different agents, through prompt engineering. These agents are required to accomplish their own tasks during the subsequent memory backtracking process. For the convenience, we list the roles of each agent as follows. More details can be found in the appendix.
Captioning Agent. Given video clip and caption instruction used guide the focus, this agent with an MLLM [2] can provide text description of the clip that meets the requirements. Its function to serve as the "eye" of the entire system. By converting visual information into text content, enables the system to perceive the video and extract memories.
Localization Agent. Given the current memory of the video (including text descriptions of various time periods) and the question, this agent, with the powerful text comprehension ability of the LLM [15] at its core, can provide a specified number of time periods that most relevant to the question. This agent enables the system to filter out interfering memories and delve deeper into relevant memories, so to achieve accurate and efficient question answering.
Instruction Agent. Given the current memory of the video, the relevant time period of interest from the localization agent, and the question, this agent can comprehensively understand the current memory, analyze the question-related key information that is missing in the given time period, and provide a guiding caption instruction. This instruction then enables the captioning agent to further delve into the memory, so as to supplement the question- related information.
Answering Agent. Given the current memory of the video and the question, this agent can, through in-depth reasoning and thinking, determine whether can strictly and objectively provide a confident answer to the question based on the current memory. If it can answer, it will provide the answer. If it cannot answer, it will output an unconfident flag. This agent should not only be capable of answering questions but also tasked with deciding whether to further explore the memory.

---

2.3 Iterative Backtracking Mechanism
Although our proposed hierarchical memory structure achieves multi-level and comprehensive information coverage of the entire long video, it is obvious that if our system is simply built on the complete memories, will undoubtedly introduce extremely high computational and storage costs, and is also likely to exceed the context limit of the LLM [15]. In addition, considering that users' questions usually focus on key time periods, a large amount of irrelevant deep memory will also turn into interfering information that cannot be ignored, which affects the system's 's performance.
Therefore, we propose a novel iterative backtracking mechanism. Through an agent-driven iterative loop, we continuously update the current memory initialized by sparse coarse memory, so as to dynamically explore the question-related memory terms of both breadth and depth. This mechanism emulates the human recollection process and achieves a comprehensive search and integration of information relevant to the question with relatively low resource cost.
Algorithm 1 The Iterative Backtracking Mechanism
Input: The video V. question Q, captioning agent CapAGT, localization agent LocAGT, instruction agent InsAGT, answering agent AnsAGT and the specified temporal scopes Te, T,, Tus corresponding to coarse, fine, and ultra-fine memory.
1:
Implement sparse coarse memory initialization to obtain initial current memory list C.M.
2: Initialize relevant set of time periods S.z 0-
3: Obtain the response based on the current memory H- AnsAGT Q).
4: while R is not confident do
5: Locate the single most ceestion-reeevant time period = Aoe \ Sr,, Q) not in S...
6: Add this time period the relevant Srt, i.c., S.t + Srt (t).
7:
Analyze missing question-key info and provide instruction prompt p = AGT
8:
Obtain the video clip Vi corresponding to this period from the video V.
9:
Divide V, into short (lips(((', V='))== by Ta. [t] = To = Ta = Ty, |4| T, = Ta = Tag.
10:
Obtain the updated curtent-depth memory of this time period me CapAGT(V..P))
11:
Ohtain the deeper memories of this time period {mi]--)
(CopAGT(Vi, k))
12:
Undate CM: CMM +- CM U [(t,, mc)} U {(t), m')) | = 1.......) L.}.
13:
Ohtain the response based on the updated current memory R AGT(
14:
end while
Output: The final response It with confident answer to the question Q.
Sparse Coarse Memory Initialization. We maintain dynamically updated current memory list CM, which is initialized with sparse coarse memory. Specifically, for a single video V [ we first set relatively large temporal scope T. corresponding to the coarse memory. Employing our captioning agent as the video captioner, we can obtain K, text descriptions {m+ knkk. Ke
{ ]======acooo according to
Eq. 1, where Ke -
1. The time period th corresponding to each m in the video are also recorded.
Then, we initialize the current memory list and obtain CM = {(th )= However, as stated previously, most of the memories are irrelevant to the questions. Therefore, we also adopt a sparse strategy. Specifically, for a question, we utilize the localization agent to find several most relevant time periods in CMiisst, forming a time period set S, Subsequently, the updated current memory list can be obtained through filtering according to the time index by C'Msinit = C'Minst StS
Ke k=1
Question-guided Memory Exploration in Depth and Breadth. Generally speaking, just as it is difficult for humans to determine the details of past events based on vague memories, relying solely on the coarse-grained initial current memory list is not sufficient provide a confident answer to the question. Therefore, our system is designed to conduct memory exploration in terms of depth (the detail level of the same time period) and breadth (different time periods) further search for comprehensive question-relevant information. Specifically, given a question and the current memory list, we first let the localization agent locate the single most question-relevant time period. Then, we let the instruction agent analyze what question-related key information is lacking in the current text description of this time period, so as to provide a caption instruction. Finally, we instruct the caption agent to generate two types of text descriptions based on the video clip of this time period and the caption instruction. On one hand, it regenerates new text descriptions for the entire video clip (updating the current-depth memory). On the other hand, it divides the video clip into more short segments and generates text descriptions for each of them (exploring deeper memory). These descriptions will be used to update the current memory list. By iteratively executing this procedure,
5

---

Pepperal
Iemporal Algg
Porcoption
11:00
and 18S0
1140
Eveel Dontest
Exent Ractitiation
Esent Recasstr action Acctontar 2007
Figure 3: Question types and examples in the EgoMem benchmark. We design six question types to comprehensively assess the model's 's understanding of cross-temporal events in long videos. Addi- tionally, a detail-perception task evaluates its ability to capture instantaneous visual features. All the questions are annotated manually, accompanied by sufficient evidence descriptions.
we obtain memories of different time periods augmented with question-relevant text descriptions. In other words, memory exploration terms of both depth and breadth in achieved.
Agent-driven Iterative Loop. To efficiently answer questions regarding long videos and continuously collect question-relevant deep memories, we propose an iterative loop driven by multiple agents. The aforementioned two procedures, serving as the core functions of our system, are utilized in the agent-drived iterative loop. The specific algorithm process is shown in Alg. 1. In simple terms, we first obtain a memory list that is strongly related to the question through the proposed sparse coarse memory initialization method. Then, based on this memory list, we carry out multiple iterative memory explorations guided by the question, so to fully search for and integrate the question- related detailed information across different time periods in the video. The iterative loop will stop only when the answering agent believes that the contained information in the updated memory list allows it to confidently answer the question. Then, it will output the corresponding confident answer. In practice, we set the maximum number of iterations to prevent getting trapped in information search chain that causes timeouts. Through the cooperation of multiple agents, this iterative loop can efficiently achieve memory backtracking related to the question, as provide confident answers by searching and integrating objective clues as much as possible.
3 EgoMem Benchmark
We construct a new benchmark for ultra-long video understanding, namely EgoMem, aiming to measure the ability to model instantaneous (detail perception) and continuous (event understanding) memory of long videos. Based on the video resources of EgoLife [51], we manually annotate question-answer pairs that particularly focus on the understanding of cross-temporal events and the perception of instantaneous visual features for each day's long video. As shown in Fig. 3, for the event understanding, we design six different question types to conduct a comprehensive and effective evaluation of the model's performance in real sense, and to avoid shortcuts. In addition, we manually annotate questions about the subtle visual features in those instantaneous time segments to assess whether the model can effectively cover detailed information. It contains 42 videos with an average duration of 6.33 hours and 504 questions. More details can be found in the appendix.
4 Experiments
Implementation Details. Our VideoLucy, as an agent-based system, only requires one LLM and MLLM for text comprehension and vision captioning. Unlike most other methods that call the APIs of expensive closed-sources proprictary models [I, 42], we consistently use the open-sourced models Qwen-2.5-VL-7B [2] and DeepSeck-RI [15] to ensure the reproducibility of the results and the low cost, if not specially stated. The temporal scopes Tc, Ty. Tuf have distinct settings for different video benchmarks. We provide more details, especially the agent prompts, in the appendix.
Evaluation Benchmarks and Metrics. Following common practices, we conduct experiments mainly on three existing long video benchmarks. MIVU [67] is a comprehensive benchmark designed to evaluate both global and local video understanding, encompassing diverse set of nine tasks. It includes videos ranging from 3 minutes to 2 hours in duration. Video-MME [11] contains

---

Method
ER
EU
KIR
TG
Rea
Sum
Overall
Method
M-Avg
Proprictary Models
Proprietary Models
Gemini 1.5 [42]
32.1
30.9
39.3
31.8
27.0
32.8
33.1
GPT-4V [1]
49.2
GLM-4V-plus-011 [13]
46.2
47.8
54.1
42.7
46.5
37.9
48.7
GPT-4o [1]
64.6
GPT-4o-20241120 [1]
48.9
49.5
48.1
40,9
50.3
50.0
48.9
Open-Sourced MLLMs
Leading Open-sourced MLLMs
Vidco-CCAM-144 [10]
63.1
TimeMarker-8B [5]
42.8
39.1
34.9
38.7
38.2
48.8
41.3
Video-XL-7B [37]
64.9
VideoLLaMA3-78 [58]
45.8
42.4
47.8
35.9
45.8
36,2
45.3
I.L.VAOVV-2B
66.4
I [7]
43.8
42.0
42.1
36.8
51.0
37.9
43.6
LinVT-7B [12]
68.9
Qwen2-VL-72B [13]
38.0
41.1
38.3
41.4
46.5
46.6
41.3
Aria-25B [19]
70.6
ReTakee7B [45]
49.8
46.2
52.9
45.0
45.8
27.6
47.8
VideoChatt-Fash-71 [24]
51.1
49.0
48.5
34.5
48.2
Oryx-1.5-32B [30]
72.3
AdaReTaKe=72B [46]
53.0
50.7
62.2
45.5
54.7
37,9
53.3
Videol aMA3-7B [58]
73.0
MaccChat--Flah--21 [24]
74.7
Agent-based Systems
VideoAgent [47]
28.0
30.3
28.0
29.3
28.0
36.4
29.3
Agent-based Systems
VideoTree [48]
30.3
25.1
26.5
27.7
31.9
25.5
28.8
Video'rree [48]
60.4
MemVid [56]
53.4
40.6
46.3
34,9
43.2
28.1
44.4
Video-RAG-7 [31]
72.4
VCA [52]
43.7
40.7
37.8
46.2
27.3
41.3
VidcoMind-7B [29]
64.4
VideoLucy
54.3
59.8
75.6
51.7
55.9
49.1
58.8
Videolucc
76.1
Table 2: Comparison results LVBench. Most are from the latest leading results of the official leaderboard (updated 20250515), and the results of the agent systems are from the corresponding papers.
Table 3: MLVU comparison. The latest leading results of the official leaderboard (updated 20250515) or respective papers.
2,700 manually annotated questions of 900 distinct videos with different durations: short (< 2min), medium (4min-15min), and long (30min-60min), We report its results under the without subtitle setting. LVBench [44] is designed for ultra-long video understanding, containing 1,492 questions of 99 videos, with an average duration of 4,101 seconds. It features diverse set of 6 tasks, all supported by high-quality human annotations. Additionally, our proposed EgoMem benchmark is utilized for further performance comparisons. As the default setting, we accuracy the evaluation metric.
4.1 Main Comparison with Other Methods
Table 1: Video-MME (w/o subs) comparison.
Comparison on Video-MME. We compare our VideoLucy with latest leading methods Video- MME [11] in Tab. 1. We classify the previous methods into three categories, the proprictary models, the open-source MLLMs trained with numerous image and video data, and the agent- based systems with eye-catching zero-shot capa- bilities. As it shows, our VideoLucy achieves a far-surpassing performance compared with other agent systems, with average accuracy 8.5% higher than that of the previous best, Mem Vid. In addi- tion, our VideoLucy stands out in terms of long video understanding. Even though VidcoLucy uses a 7B MLLM, still achieves the best perfor- mance among all open-sourced MILMs in this aspect, outperforming the best model Adaretake- 72B by 1.8%, and is par with the leading commercial proprietary model Gemini 1.5 pro.
The latest leading results of the official leader- board (updated 20250515) or respective papers.
Method
Short Medium Long
Avg.
Proprietary Models
Gemini 1.5 Flash [42]
78.8
68.8
61.1
70.3
NPPT-40-22240513 [1]
80.0
70.3
65.3
71.9
Gemini 1.5 pro [42]
81.7
74.3
67.4
75.0
Leading Open soureed MLLMS MLLMS
Qwen2-VL-72B [43]
80.1
71.3
62.2
71.2
AdaReTake-7722 [46]
80.6
74.9
65.0
73.5
IntemVL22.5-228
82.8
70.9
62.6
72.1
LLaVA-OneVisioon- [18]
76.7
62.2
60.0
66.3
LLa Vidco-72B [63]
81.4
68.9
61.5
70.6
VideoChat-Flash--B [24]
78.0
67.8
55.6
65.3
VideoL_ [58]
63.7
LiveCC-77B [4]
74.8
63.9
53.7
64.1
ViLAMP-7B [8]
78.9
65.8
57.8
67.5
LinVT-7B [12]
79.0
71.6
63.2
70.3
Agent-based Systems
VidcoAgent [9]
46.4
VideoTree [48]
67.8
59.9
54.2
60.6
DrVideo [32]
51.7
Comparison on I.Bench. The outstanding abil- ity of our VidcoLucy has been further signifi- cantly demonstrated the ultra-long benchmark LVBench [44]. As shown in Tab. 2, our Vide-
MemVid [56]
73.9
63.1
55,0
64.0
Video-RACE RAG-7B
66.4
60.2
62.1
VCA [52]
56.3
VideoLucy
78.6
72.1
66.8
72.5
oLucy achieves performance far exceeding the previous best methods across all metrics. Its overall performance is 5.5% higher than the latest best method [46] on the official leaderboard, achieving 58.8% accuracy. In addition, it performs particularly outstandingly in key information retrieval (KIR), achieving an accuracy rate of 75.6%, significantly surpassing all previous models.
Comparison on MLVU. We also evaluate the performance of our Videolucy on the comprehensive benchmark MLVU [67], which covers spanning video durations. As shown in Tab. 3, compared with the latest leading methods the official leaderboard, VideoLucy still achieves better overall
7

---

results. This indicates that our VideoLucy has excellent adaptability to videos of various durations and can effectively meet the situation where the video duration is unknown in practical applications.
Comparison on EgoMem. We also carried out a performance evaluation comparison on our proposed EgoMem benchmark. As shown in Tab. 4, we tested series of the latest leading open-sourced video MLLMs. It can be seen that existing MLLMs perform poorly, and they are seriously lacking in cross-temporal understanding and detail perception abilities in the case of extremely long videos. Their evaluation results [43, 2, 7, 69] are only slightly better than the results of random guessing. However, our VideoLucy. can better understand cross- tross-temporal and detailed information of extremely long videos through the memory backtracking mechanism. Compared with other methods, it achieves the best performance on EgoMem, with an overall accuracy rate of 56.7%, which is 10.3% higher than that of the latest ultra-long video understanding model, VideoChat- [24].
Table 4: Evaluation results on our proposed EgoMem benchmark.
Method
Event
Detail
Avg
Leading Open-sourced MS
Qwen2-VL-7B [43]
38,9
25.8
32.3
Owen2-VL--72B [43]
38.1
27.4
32.7
Qwen2.5- [2]
36.5
27.8
32.2
Qwen2,5-VLL-72H [2]
37.7
30.2
33.9
IntemmVL2,558B [7]
33.7
34,5
34.1
IntemmV1.2.5-788 [7]
32.7
38.3
35.5
4.2 Ablation and Analysis
IntemVV1338BB [69]
28.2
38.9
33.6
IntemVLL3-78B [69]
27.4
35,2
31.3
Nedde-in-AA-iiee-Haayal Video-Haystack. To further explore the model's ability to perceive the detailed instantaneous events in long videos, following the common practices [24, 62, 65], we con- duct an evaluation experiment named "Needle-in-A. Video- Haystack". Specifically, we randomly select 10 long videos with durations ranging from 400s to 4000s from the existing benchmarks [67. 44]. Then, we insert 10s short video clips (needles) from the Internet at five arbitrary timestamps throughout each long video, from the beginning to the end. Then we feed the entire long video into the model and con- duct question-answering test regarding the content of these short clips. There are 4 questions for each clip, forming 20 questions for one long video. As shown in Fig. 4, the performance of our VideoLucy is significantly better than that of the existing leading models, and its results are almost unaffected by the video length. This indicates that our VideoLucy has a very strong ability to search for question-relevant details in long video understanding, More details are shown in the appendix.
VidcoChat-Flash-72 [24]
44.8
48.0
46.4
Agent-bused Systems
VidcoAgent [47]
28.6
34.5
31.5
VidenTree [48]
30.2
36.1
33.1
VideoLucy
58.7
54.8
56.7
100
20
8.1
8.0
7.5K1
7,005
7.9
2
3.7-
08K
12K
2,0K
2,0K
3,2K
7.6
U.
Video Dazadion
Figure Results on the "Needle-in-A-Vice Haystack" evaluation with 10 long videos.
Figure 5: Evaluation information richness and relevance in memory backtracking of VideoLucy.
Information Richness and Relevance in Back- tracking. We experiment to explore the infor- mation richness of memories at different levels during memory backtracking and their relevance to the given questions. Specifically, we first se- lect the test samples in Video-MME [11] and IVBench [44] that enter the deepest memory of VideoLucy. Then, we evaluate the information richness by calculating the average value of the Shannon entropy of the text descriptions of mem- ories at all levels for each test sample. In addition, through prompt engineering, we make an LLM [35] serve as a relevance evaluator, enabling it to assess the relevance between the given text description and the question. As shown in Fig. 5, during the backtracking process, both the richness and relevance increase continuously, verifying the effectiveness of method. More details are shown in the appendix.
68
668
66
900
64
622
62 Aecta 60
65 Gel
50
56
6
Elflec of memory depth
[t)
Effle:
Docration number
Figure 6: Effects of different memory depth and maximum iteration number. We evaluate the per- formance on Video-MME long split (w/ subs).
Effects of Different Components. We conduct an experiment on the Video-MME [11] long split to explore the effectiveness of memories at various levels on the performance of the model. The
8

---

18:09
24.8
Briefcase deliver
Hotell confrontation
Drug implantation
Hotel confrontation
Beaten in the abdomen
Superpower activation
Q Question: At the beginning, what is the direct trigger for the first rapid activation of Lucy's superpowers? W hat series of events has Lucy experienced before that? Please list this series of events step by step according to the storyline.
VideoLucy
Reasoning
Tolking Into Deep Metory:: Search [1400
1600L0n)
Search (800, Co)
Swarch (902) 500
Soarch (200 fi. 400.0e)
samto switin tome popiiia y dine
be braataue can tn ciac 1ide an big un pl madia
paric
cott confrrttatie
eccalasion thar
the remmoleen sabre vol Thess canas aie uniccaes deartt org rigper Lucn's peroowe catic procaciig earra menticie welddedd carrdie
parict
imglaraaion
abdieaa coring
the ucstaca that supares we beating tippeing ug acvess why ol 1200 L0000 conte
tuey to forse prrertatto an new Lirde standing inwective envert emabile moterdoen and prene bnttuae
coneai luc si, deitger coubis hetted whyy the trlelle
which divechy
Confident Answeri
The direet trigger for the first ripid artivation of Lugy's superpowers
viulent beating in dimly emvironment that caused the rupture of the this, Lucy was coerced into delivering a locked briefcase (containing CPH)) Mr. J ang a a cowbay hat who offered her pay ment. The sorees of events included: 1) Be entrusted to deliver the briefcase; 2) Confront a group of people in the hotel and be implanted with CPH4; 3) Subsequeetty suffer from violent beating, caustng the CPH4 to rupture.
CPI implanted
abdomen.
man
Owwen2255 V2B-
IntoraVL LL--MVL-3-888 BB
Video Chart F Flash
Gannaay Bo
sunnig Stubill
amas Not
a falan Batker
[kdiar) mippis big Lel. Le ditrched Eduepandly Lay's
dhe begar Lay conse to bulkinng acseppaed be aaania
dark coom. Betine this Locy has apediaod a soties of eyent incluilin berg cabupped benn boon io sword loodist. bedag wihjected In variesa limbs and experiments, and niaaly beleg rescued a group of people.
pagge
Areond Mack Itarie 1ik time. saperpooons
Intt whols eisily
thata Tai me quada
gras quadic balinted
and fur pnnked stagle le belig to dede mosil tut attadaanvi
aliepyy Mtdldd beteceii moRA)
Figure 7: Qualitative comparison of event understanding in long videos. Compared with existing leading video MLLMs [2, 69, 24], our VideoLucy stands out in capturing and integrating cross- temporal events in long videos, along with explainable and comprehensive reasoning process.
experiment is divided into four groups: model answering with only a rough video summary; a model relying on coarse-memory text; model with fine-grained memory access; and model with ultra-fine-gratte -reeggr memory access. As shown in Fig. 6 (a). delving into memories at all levels can effectively improve the model performance, and the optimal result is achieved when delving into the frame-level ultra-fine memory. In addition, we explore the impact of the maximum number of iterations in the loop. As shown in Fig. 6 (b), the best performance is achieved when the number of iterations is set to 5, which is set as our default value.
4.3 Qualitative Comparison
A Biamese with: blue ayns and at
We conduct a qualitative comparison exper- iment using a long video of the movie Lucy as the test material. The experiment consists of two tasks. In the first task (Fig. 7), mod- els 12, 69, 24] are required to address ques- tions on cross-temporal event understanding. As it shows, our Videolucy accurately cap- tures inter-temporal event relationships, out- performing the other three models that often provide inaccurate or incomplete answers. In the second task (Fig. 8), models are asked to answer questions about the detailed infor- mation of momentary segments with fleeting events. As we can see, our Videol uuyy per- forms well in extracting and presenting fine
There's scene w hhree as the doctor is removing ing the CPH4 from Lucy's abdomen, she's on the phone with her mom. They mention a cat during the call. So, what's the like?
Question:
VideoLucy
Ressoning Process:
Casca Mievry Fiker (Tly The Thee Peborh (D000) 230 Et (18901 EC, Resene pondle
This spucitaaly cactics bez alow aurtaaaie dacasiao perani casaadio malh Tam Fand 22 weth up fi beop Menwey haking "Tiie find poke so0nrt Senoee "The quodin speat ofly weloww so phone
bewen Lazy be cadde darin madkal procetes (2it meanidd waich alana vit Thms Poatd matel Oncepiic metee diseninng vmirry avaccsso and faeiyy via de pi does act eqgi city
dasupe
Confident Answer:
SCraldmmn The mnooe The cat Slamose with eyes and a broken
eypictly cate the vinaa roal Vinking Twe on nox ant the sclrguee tine deldd no 2190 nette quatt Summs with nu add arsses momppmeet
candios
death ecpliid
bi
withet tileman")) ").
InternVL.. LL3-778
VideoChat-Fiaan
details, while the other two models strug- gle capture nuanced information, yielding poor results. Additionally, for each question in both tasks, our VideoLucy not only pro- vides the correct answer but also shows the
The vidm divs not prodde specifc detalls regunting sppearance cat. 11 a cheing phane dacett
ch is described as having white lace and black body.
dexcess
2a partodia s035.
Figure 8: Qualitative comparison of detail perception in long videos. VideoLucy enjoys great superiority.
step-by-step process of how it arrives at that conclusion. This transparency helps users understand the model's 's decision decision-making logic in practice, enhancing result interpretability and trust.
Related Work
MILLMs for Long Video Understanding. Existing video MILMS [22, 21, 26, 60, 36, 23] are generally built upon conventional image-based VLM frameworks [27, 68, 20J. They employ visual
9

---

encoder to convert frames into tokens, which are then projected into an LLM's semantic space to support tasks such as video captioning, QA, and reasoning. For long videos, two main trends distinguish video MLLMs from standard VL.S.
First, frame sampling has gained attention as critical component. Since processing all frames in long videos is computationally prohibitive, the focus is on selecting key frames relevant to the query or task. Several studies [41, 16, 55, 54, 53, 33, 40, 61] adopt attention-based mechanisms to model spatio- temporal relationships among frames. For example, [41] proposes Adaptive Keyframe Sampling (AKS), which optimizes both relevance to the prompt and video coverage under a fixed token budget, thereby improving QA performance and underscoring the value of information pre- filtering.
Second, token compression has emerged to address the large number of visual tokens produced by long videos, which can exceed I.IM capacity and increase computational overhead. Researchers [25, 10, 28, 38, 49, 6, 57] have developed methods to compress tokens while preserving essential information. For instance, VideoChat-Flan [24] introduces a hierarchical token compression approach combined with multistage learning strategy, achieving strong performance under high compression rates.
In addition, some recent works have introduced memory-augmented methods. LangRepo [17] pioneers a long-video understanding framework that maintains a structured language repository for LLMs. It iteratively updates multi-scale video chunks and prunes redundant text via write/read operations. Similarly, TTM [34] propose an efficient memory- augmented Transformer inspired by Neural Turing Machines. It uses an external memory to compress historical frames into compact tokens, limiting computational complexity as each new frame interacts only with memory tokens.
Nevertheless, these conventional video MILMs still struggle to effectively represent ultra-long videos. Their reliance on sparse frame sampling often results in significant information loss and suboptimal responses in real-world scenarios.
Agent-based Systems for Long Video Understanding. Recent studies [50, 14, 3, 32, 9, 47, 66, 48] have introduced agent-based systems as promising solution for long video understanding. Unlike traditional video MLLMs, which struggle with lengthy visual inputs, these systems utilize LLMs for reasoning, planning, and memory management, By employing LLMs to orchestrate MILMs, they decompose long videos into short-video subtasks, iteratively retrieving and integrating question- relevant information. This approach mitigates length constraints and enhances comprehension.
For instance, VideoAgent [47] employs an LLM as a central agent that iteratively locates and consolidates key information using vision-language models. DrVideo [32] transforms long videos into text documents and applies agent-based loops to retrieve key frames and augment data. VrdeoTree [48] builds a query-adaptive hierarchical video representation via iterative keyframe refinement and tree- based aggregation, enabling efficient LLM reasoning.
However, noted, these systems typically reason over individual frames, overlooking temporal context, and often adopt sparse sampling to reduce captioning costs- potentially missing critical information. These limitations highlight the need for improvement, motivating our Videol.cy.
Conclusion
We introduced VideoLucy, a novel deep memory backtracking framework. Inspired by human recol- lection processes, Videol Jucy employs a hierarchical memory structure with progressive granularity and an agent-based iterative backtracking mechanism. This design enables the system to dynamically and comprehensively mine question-relevant information from the entire video, starting from a coarse overview and progressively delving into finer spatiotemporal details, thereby achieving robust temporal understanding and unprecedented fine-grained perception. Furthermore, we presented EgoMem, new benchmark for evaluating temporal and fine-grained understanding extremely long videos. Extensive experiments demonstrate that VideoLucy achieves state-of-the-art performance across multiple benchmarks, significantly outperforming previous methods and even surpassing pow- erful proprietary models. Our work validates the effectiveness of a structured. human-like memory mechanism for complex video understanding and paves the way for future research in this field.
Acknowledgments.
This work was supported by the National Natural Science Foundation of China No.U22B2053 and the Hubei Provincial Natural Science Foundation of China No. F5.2022C
10

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint Sktv:22303087774 2023.
[2] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 2025.
[3] Shiwen Cao. Zhaoxing Zhang, Junming Jiao, Juyi Qiao, Guowen Song, and Rong Shen. Meaf: Efficient agent-based video understanding framework through multimodal coarse-to-fine attention focusing. arXiv preprint arXiv:2504. 17213, 2025.
[4] Joya Chen, Ziyun Zeng, Yiqi Lin, Wei Li, Zejun Ma, and Mike Zheng Shou. Livecc: Learning video IIm with streaming speech transcription at scale. In CVPR, 2025.
[5] Shimin Chen, Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma. Timemarker: A versatile video-Ilm for long and short video understanding with superior temporal localization ability. arXiv preprint arXiv: 18211, 2024.
[6] Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang. Haotian Tang, Shang Yang. Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. arXiv preprint arXiv:2408. 10188, 2024.
[7] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Ye, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.002211, 2024.
[8] Chuanqi Cheng, Jian Guan, Wei Wu, and Rui Yan. Scaling video-language models to 10k frames via hierarchical differential distillation. In ICML, 2025.
[9] Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. Videoagent: A memory-augmente multimodal agent for video understanding. In ECCV, pages 75-92. Springer, 2024.
[10] Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal cross-attention masks for short and long videos. arXiv preprint arXiv:2408. /4023, 2024.
[11] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-mon modal lims in video analysis. In CVPR, 2025.
[12] Lishuai Gao, Yujie Zhong, Yingsen Zeng. Haoxian Tan, Dengjie Li, and Zheng Zhao. Linvt: Empower your image-level large language model to understand videos. arXiv preprint arXiv:2412.05185. 2024.
[13] Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv preprint arXiv:2406. 12793, 2024.
[14] Gabriele Goletto, Tushar Nagarajan, Giuseppe Averta, and Dima Damen. Amego: Active memory from long egocentric videos. In ECCV, pages 92-110. Springer, 2024.
[15] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12944 2025.
[16] Kai Hu, Feng Gao, Xiaohan Nie, Peng Zhou, Son Tran, Tal Neiman, Lingyun Wang, Mubarak Shah, Raffay Hamid, Bing Yin, et al. M-llm based video frame selection for efficient video understanding. arXiv preprint arXiv:2502.1 2tti::2502.19880 19680, 2025.
[17] Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S Ryoo. Language repository for long video understanding. arXiv preprint arXiv:2403.14622. 2024.
[18] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevisiion: Easy visual task transfer. arXiv preprint arXiv:2408.0333 033266 2024.

---

[19] Dongxu Li, Yudong Liu, Haoning Wu, Yue Wang, Zhiqi Shen. Bowen Qu, Xinyao Niu, Fan Zhou, Chengen Huang, Yampeng Li, al. Aria: An open multimodal native mixture-of-expeee model, arXiv preprini arXiv:2410.05993, 2024.
[20] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In ICMI., pages 19730-19742. PMLR, 2023.
[21] KunChang Li, Yinan He, Yi Wang. Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang. and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint a:Xiv:2305.06355, 2023.
[22] Kunchang Li, Yali Wang. Yinan He, Yizhuo Li, Yi Wang. Yi Liu, Zun Wang. Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In CVPR, pages 22295, 2024.
[23] Wei Li, Hehe Fan, Yongkang Wong, Mohan S Kankanhalli, and Yi Yang. Topa: Extending large language models for video understanding via text-only pre-alignment. NeurIPS, 37:5697-5738, 2024.
[24] Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng. Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Videochat-flan Hierarchical compression for long-context video modeling. arXiv preprint arXiv:2501.00577 005744 2024.
[25] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Computer Vision, pages 323-340. Springer, 2024.
[26] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. In EMNLP, pages 5971-5984, 2024.
[27] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurlPS, 10:34892-34915 2023.
[28] Xiangrui Liu, Yan Shu, Zheng Liu, Ao Li, Yang Tian, and Bo Zhao. Video-xl-pro: Re- constructive token compression for extremely long video understanding. arXiv preprint arXiv:2503. 18478, 2025.
[29] Ye Liu, Kevin Qinghong Lin, Chang Wen Chen, and Mike Zheng Shou. Videomind: A chain-of-lora agent for long video reasoning. arXiv preprint arXiv:2503. 13444, 2025.
[30] Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx milm: On-demand seatoal-temporal understanding at arbitrary resolution. In ICLR, 2025.
[31] Yongdong Luo, Xiawu Zheng, Xiao Yang, Guilin Li, Haojia Lin, Jinfa Huang, Jiayi Ji, Fei Chao, Jiebo Luo, and Rongrong Ji. Video-rag: eesuall--alinne comprehension. arXiv preprint arXiv:2411. 130993 2024.
serieval-augmenteer long video
[32] Ziyu Ma, Chenhui Gou, Hengcan Shi, Bin Sun, Shutao Li, Hamid Rezatofighi, and Jianfei Cai. Drvideo: Document retrieval based long video understanding. CVPR, 2025.
[33] Jongwoo Park, Kanchana Ranasinghe, Kumara Kahatapitiya, Wonjeong Ryu, Donghyun Kim, and Michael S Ryoo. Too many frames, not all useful: Efficient strategies for long-form video qa. arXiv preprint arXiv:2406.09330 2024.
[34] Michael S Ryoo, Keerthana Gopalakrishnan, Kumara Kahatapitiya, Ted Xiao, Kanishka Rao, Austin Stone, Yao Lu, Julian Ibarz, and Anurag Arna.. Token turing machines. In CVPR, pages 19070-19081, 2023.
[35] ByteDance Seed, Yufeng Yuan, Yu Yue, Mingxuan Wang, Xiaochen Zuo, Jiaze Chen, Lin Yan, Wenyuan Xu, Chi Zhang, Xin Liu, al. Seed-thinking-vl. 5: Advancing superb reasoning models with reinforcement learning. arXiv preprint arXiv:2504. 13914, 2025.
[36] Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spa- tiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410. 17434, 2024.
[37] Yan Shu, Zheng Liu, Peitian Zhang, Minghao Qin, Junjie Zhou, Zhengyang Liang, Tiejun Huang, and Bo Zhao. Video-xl: Extra-long vision language model for hour-scale video understanding. arXiv preprint arXiv:2409.144 14485, 2024.
12

---

[38]
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, al. Moviechat: From dense token to sparse memory for long video understanding. In CVPR, pages 18221-18232, 2024.
[39] Yucheng Suo, Fan Ma, Linchao Zhu, Tianyi Wang, Fengyun Rao, and Yi Yang. From trial to triumph: Advancing long video understanding via visual context sample scaling and self-reward alignment. arXiv preprint arXiv:2503. 20472, 2025.
[40] Reuben Tan, Ximeng Sun, Ping Hu, Jui-hsien Wang. Hanich Deilamsalchy, Bryan A Plummer, Bryan Russell, and Kate Saenko. Koala: Key condicondd long video-Ilm. In CVPR, pages 13581-13591, 2024.
[41] Xi Tang, Jihao Qiu, Lingxi Xie, Yunjie Tian, Jianbin Jiao, and Qixiang Ye. Adaptive keyframe sampling for long video understanding. In CVPR, 2025.
[42] Gemini Team, Rohan Anil, Sehastian Borgeaud, Jew.- -aan-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, al. Gemini: family of highly capable multimodal models. arXiv preprint 1rXiv:2312.11805. 2023,
[43] Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Gc, et al. Qwen2-vl: Enhancing rision-languaal model's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024.
[44] Weihan Wang, Zehai He, Wenyi Hong. Yean Cheng, Xiaohan Zhang, Ji Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. arXiv preprint arXiv:2406.086 08035, 2024.
[45] Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nie. Retake: Re- ducing temporal and knowledge redundancy for long video understanding. arXiv preprint
arXiv:2412.20504, 2024.
[46] Xiao Wang, Qingyi Si, Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nie. Adaretake: Adaptive redundancy reduction to perceive longer for video-language understanding. arXiv preprint arXiv:2503./2559, 2025.
[47] Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. In ECCV, pages 58-76. Springer, 2024.
[48] Ziyang Wang, Shoubin Yu, Elias Stengel-Eskin, Jachong Yoon, Feng Cheng, Gedas Bertasius, and Mohit Bansal. Videotree: Adaptive tree-based video representation for Ilm reasoning on long videos. In CVPR, 2025.
[49]
Yuetian Weng, Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longvlm: Efficient long video understanding via large language models. In ECCV, pages 453-470. Springer, 2024.
[50] Yuxuan Yan, Shiqi Jiang, Ting Cao, Yifan Yang, Qianqian Yang, Yuanchao Shu, Yuqing Yang, and Lili Qiu. Empowering agentic video analytics systems with video language models. arXiv preprint arXiv:2505. 0025,, 2025.
[51] Jingkang Yang, Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang, et al. Egolife: Towards egocentric life assistant. In CVPR, 2025.
[52] Zeyuan Yang, Delin Chen, Xueyang Yu, Maohao Shen, and Chuang Gan. Vca: Video curious agent for long video understanding. arXiv preprint arXin:2412.10477, 2024.
[53] Linli Yao, Haoning Wu, Kun Ouyang, Yuanxing Zhang, Caiming Xiong, Bei Chen, Xu Sun, and Junnan Li. Generative frame sampler for long video understanding. arXiv preprint arXiv:2503.091 0944, 2025.
[54] Jinhui Ye, Zihan Wang, Haosen Sun, Keshigeyan Chandrasegaran, Zane Durante, Cristobal Eyzaguirre, Yonatan Bisk, Juan Carlos Niebles, Ehsan Adeli, Li Fei-Fei, et al. Re-thinking temporal search for long-form video understanding. In CVPR, 2025.
[55] Shoubin Yu, Jaemin Cho, Prateck Yadav, and Mohit Bansal. Self-chained model for video localization and question answering. NewrIPS, 36.76749-76771, 2023.
[56] Huaying Yuan, Zheng Liu, Minhao Qin, Hongjin Qian, Y Shu, Zhicheng Dou, and Ji- Ji-Rong Wen. eenore-edhanced retrieval augmentation for long video understanding. arXiv preprint 25X3::2503.09149, 2025.
13

---

[57] Xiangyu Zeng, Kunchang Li, Chenting Wang. Xinhao Li, Tianxiang Jiang. Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang, al. Timesuite: Improving milms for long video understanding via grounded tuning. arXiv preprini arXiv:2410. 10. 19702, 2024.
[58] Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang. Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding. arXiv preprint 1rXi::2501133100 2025.
[59] Ce Zhang, Taixi Lu, Md Mohaiminul Islam. Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple Ilm framework for long-range video question wvestion-answeringg arXiv preprint arXiv:23/2.1725: 2023.
[60] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction audio-visual language model for video understanding. arXiv preprint arXiv:2306.02883 2023.
[61] Huaxin Zhang, Xiaohao Xu, Xiang Wang, Jialong Zuo, Xiaonan Huang, Changxin Gao, Shanjun Zhang, Li Yu, and Nong Sang. Holmes-vau: Towards long-term video anomaly understanding at any granularity. In CVPR, 2025.
[62] Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language vision. arXiv preprint arXiv:2406. 16852, 2024.
[63] Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with symthetic data. arXiv preprint 241002241..02713, 2024.
[64] Yue Zhao, Ishan Misra, Philipp Krähenbühl, and Rohit Girdhar. Learning video representations from large language models. In CVPR, pages 6586-6597, 2023.
[65] Zijia Zhao, Haoyu Lu, Yuqi Huo, Yifan Du, Tongtian Yue, Longteng Guo, Bingning Wang, Weipeng Chen, and Jing Liu. Needle in a video haystack: A scalable synthetic framework for benchmarking video milms. arXiv e-prints, pages arXiv-2406, 2024.
[66] Zhuo Zhi, Qiangqiang Wu, Wenbo Li, Yinchuan Li, Kun Shao, Kaiwen Zhou, et al. Videoagent2: Enhancing the llm-based agent system for long-form video understanding by uncertainty-aware cot. arXiv preprint (rXiv:25040047/1 2025.
[67] Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang. and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.04266 2024.
[68] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[69] Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims made in the abstract and introduction our paper accurately reflect our paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We conduct an in-depth discussion of the limitations of our work, which can found Sec. A.7 of the appendix.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification:
Our work focuses on computer vision applications, not including theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have fully provided all the details of our proposed methods in the main text and supplementary materials, to guarantee the reproducibility of our paper's 's main experimental results. Meanwhile, we will open-source our work after our paper is accepted. These details and open-source initiatives will ensure the reproducibility of our work and make significant contribution to the entire community.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the partieular way they provide for reproducibiliity In the case of closed-source models, it may be that access to the model is limited in
16

---

some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: To ensure the proper use the technology, we would not provide open access to our data and code during the paper submission process. We provide the testing logs in the supplementary materials to ensure the authenticity and validity of our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
cesiionooicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ /nipp..c..... hpppuulccguuddeieeee /Caeeee///memamiiaa citomisssii caiioioPpccyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our paper specifies all the training and test details necessary to understand the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Due to long video understanding technology is comparably computational expensive, the error bars not reported in the paper like nearly all related work. We provide the testing logs in the supplementary materials to ensure the authenticity and validity of our results.
Guidelines:
17

---

The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our paper provides sufficient information on the computer resources needed to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httee:////eurree /tt.//nurrp..//e wctppblcc//eeeeeeceeeeeeeeeeeeee
Answer: [Yes]
Justification: The research conducted in the paper complies with the NeurlPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
18

---

Justification:
Our paper discusses both potential positive societal impacts and negative societal impacts of the work in the supplementary materials.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper poses no such risks
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-us should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The creators and original owners of assets used the paper are properly credited, and the license and terms of use are properly respected.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
19

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thoode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
To ensure proper use, we would not release our assets at submission time.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the datasct/cod//model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing research with human subjects.
Guidelines:
20

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [Yes]
This paper focuses on agent-based systems for long video understanding, which, as is customary, include an LLM as their core. We elaborate on the details regarding the use of LLMs our paper.
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: tttpps://neuree /neuripss.....// for what should or should be described.
/LLM)
21

---

A Appendix
In this appendix, we elaborate on the following:
Technical Details. Specifically, we demonstrate the distinct settings of our VideoLucy for different video benchmarks with varying video length distributions. These settings include the configuration of memory temporal ranges at all levels, frame sampling rate settings, etc. On the other hand, describe the carefully designed prompt details for different agents.
EgoMem Benchmark Details. We provide comprehensive elaboration the meticulous and step- by-step process of manual annotation, as well as the detailed annotation content of question-answer pairs across each representative category.
Ablation Study Details. We elaborate on the specific experimental details of the two ablation studies in the main text. On one hand, for the Needle-in-A VViei----aasteeee sources of the 10 selected long videos and present the "needles" we constructed, which consist of five short videos and 20 question-answer pairs. On the other hand, for the Information Richness and Relevance in Backtracking experiment, we display the code for information entropy calculation and the prompts designed for the relevance evaluator.
experiment, we explain the
Additional Experiments. We evaluate the performance of our VideoLucy when using different MLLMs and LLMs to validate the excellent adaptability of our framework.
Failure Case Analyses. We present and analyze additional specific failure cases of VideoLucy's performance in long video understanding.
Broader Impacts. We discuss the broader societal impacts of long video understanding technology.
Limitation. We discuss the limitation of our Videol framework.
Inference Time. We discuss the inference time of our Videolucy framework.
A.1 Technical Details
Distinct Settings for Different Benchmarks
During the sparse coarse memory initialization of VideoLucy, for all benchmarks, we uniformly instruct the localization agent to first identify the three most relevant time periods. Additionally, considering that scenes with event continuity may be split into multiple periods, we adopt a neighbor- hood expansion strategy: the two time periods adjacent to a period provided by the agent are also included in the relevant time periods, As a result, the number of time periods in the sparse coarse memory ranges from three to nine. Additionally, the distinct settings are described as follows:
MLVU. Since MLVU is comprehensive video understanding benchmark with wide range of video durations, we divided it by video length: 0-600s (short), 600-12000 12-0s (medium). 1200-3600s (long), and >3600s (extra-long). Temporal scopes for memory: Short: T. = 30s, T, 5s. Medium: T. 60s, F 5s, Long: To = 100s, T' = 10s, Extra-long: T. = 200s, T3 = 10s. All use Tuf = 1s. The frame sampling rate for coarse-grained memory 1 FPS, while the others are 2 FPS.
VidEO-MME. This benchmark has been divided into short, medium, and long splits by the original authors. Following convention, adopt the official default splits. Temporal scopes for memory: Short: T, 58, T3 = 1s, Medium: T. 50s, Ty = 58, Long: T- = 100s, T 10s. All use Tuf = 1s. For medium and long videos, the frame sampling rate of coarse-grained memory is 1 FPS, while the other two are 2 FPS. For short videos, the frame sampling rate of coarse-grained memory is 2 FPS, and the other two are FPS.
LVBennh. It is a benchmark specifically designed for long video understanding, yet it still encom- passes relatively wide range of video durations. We also divide it by video length: 1800-3600s (short), 3600-5400s (medium), and >5400s (long). Temporal scopes for memory: Short: T. = 100s, T, 10s, Medium: T.- 150s, T₃ 10s, Long: T. = 200s, T, 10s. All use Tuf 1s. The frame sampling rate for coarse-grand -grrined memory is FPS, while the others are 2 FPS.
EgoMem. Since each video in this benchmark is extremely long (with average duration of 6.33 hours) and their lengths are roughly uniform, we do not perform any division on this benehmark. For each video, the temporal scopes are set as follows: coarse-grained memory (T: 800s), fine-grained
22

---

The following provides rough description of what's shown in the video during different time periods: [Current Memory List (Time Period Description) Now, a question has been raised regarding this video. [Question]
Please read the given video content descriptions and the question in depth. Since most of these descriptions are rather rough and some detailed information is lost, my task is to try my best to find the time periods related to the given question, and then provide more detailed descriptions of the video content of these time periods.
In order to assist me in completing my task, your task is to:
Based on the provided rough video descriptions, determine whether the given question allows me provide a more confident answer by further observing the video content of three time periods.
If so, you should find out the time periods related to the question as much as possible and provide these relevant time periods so that I can review the content information of these video segments again to obtain more information and answer the question better.
For example, since there is no need for overall understanding of large video segments, the following questions can obtain more accurate answers by re-observing the video segments of three time periods: (i) What color is Putin's tie between the interview with Antony Blinkoen and interview with Marie Yovanovitch?
(ii) How does the goalkeeper prevent Liverpool's 's shot from scoring at 81:38 in the video? (iii) Who smashes the magic mirror?
On the contrary, for example, because an overall understanding of large video segments is required, it is difficult to obtain more accurate answers to the following questions by merely observing two video segments:
() What happens in the second half of the game?
(ii) What is the video about?
(ii) Which places has the protagonist of this video been to in total?
You should output in a strictly standardized dictionary format containing three key-value pairs:
"Flag": A bool. If you are very confident that you can provide the time periods according to the above requirements, set it as Truc. Otherwise, set it False.
"Time Period": A list. If "Flag" is True, fill in the list with the most relevant three time periods, in the tuple format (start time, end time). If "Flag" is False, fill in "No Time Periods." "
"Reason": A String. Show me your reasons for the time periods you provided.
Table 5:
The designed prompts for the localization agent in the sparse coarse memory initialization stage. Given the current memory list and the question, obtain the most relevant three time periods.
memory (T3 80s), and altra-fine-graine memory (Tuf 8s). The frame sampling rates are respectively set to 0.25 FPS, 0.5 FPS, and 1 FPS for the three memory types.
Agent Prompts.
Captioning Agent. The primary function of this agent is to serve as the system's "eye," capable of describing given video clip according to the requirements of specified instructions. For all coarse memory extraction, which is unrelated to the question, the instruction prompt is uniformly set as: "Please observe and understand the given video carefully. Describe all the details of this video as comprehensively as possible in a smooth and coherent passage. Do not omit any details or prominent information. addition, if there are any texts, subtitles, text overlays, or voice-overs in the video, you must explicitly and in detail describe them." For memory extraction at other levels, the instruction prompts are dynamically generated by the instruction agent based on the question.
Localization Agent. The main function of this agent is to locate the temporal segments in the current memory (including text descriptions of various time periods) that are relevant to the question. This agent is frequently used in sparse coarse memory initialization to find the three most question-relevant temporal segments. Considering that some questions require a comprehensive understanding of the entire video for answering, we have designed additional functionalities for this agent during the initialization process, as follows: if the given question can be answered by re-reviewing several temporal segments, these segments will be provided; if not, no sparse initialization will be performed. The prompt of this agent in the initialization phase is shown in Tab. 5. Additionally, this agent also used to search for the single most question-relevam and interesting temporal segment in the current memory list during the memory backtracking process, after which the subsequent instruction agent will generate corresponding instructions for retrieving supplementary information. To minimize LLM calls, we merged these two processes into one, enabling the functions of the above two agents to be achieved simultancously through single designed prompt, as shown in Tab. 6.
23

---

There is currendy video with total duration of /video length) seconds. The following gives a general description what shown in the video during certain time periods: [Current Memory List (Time Period + Description) Now, a question has been raised regarding this video. (Question)
Please read the given video content descriptions and the question in depth. You do not need to answer this question. Your first task is to identify, based on the video content in each time period, the single time period that is most relevant to the question and that you think requires further elaboration of its video content details to make the answer to this question more explicit.
Notably. you need to select the most relevant one from the time periods other than the following time periods: (Already Searched Time Periods)
In addition, assume there is now a caption model that can describe given video according to your instruction. Your second task is to consider what detailed content in the video of the time period you have selected you want the model to focus describing, and provide your instruction. For example, assume that the entire video segment about an offensive play a certain football game, and you want to focus on the passing situation of the football during this offensive play. The instruction you give to the model could be:
Please observe all the details in this video very carcfully and provide detailed and objective description of what is shown the video. If this video is about an offensive play in a football match, you should focus particularly on the passing situation of the football during this offensive play.
Note that you should organize your instruction by referring to the language expressions in the above example. You should output in strictly standardized dictionary format containing three key-value pairs: "Time Period": A list. Fill with the single most relevant period, in the tuple format (start time, end time). "Instruction": A String. This string must be enclosed in double quotes. Show me the instruction you want to give to the caption model for the second task.
"Reason": : A String. This string must be enclosed in double quotes. Show me your reasons for the time period and instruction you provided.
Table 6: The designed prompts for the localization and instruction agents in the memory backtracking stage. Given the current memory list and the question, we obtain the single most relevant period and corresponding instruction.
Instruction Agent. Given the current memory list and a specified time period, this agent generates an instruction for the captioning agent to obtain question-relevant supplementary information. This agent can be merged with the localization agent, and their combined prompt is shown in Tab. 6.
A Answering Agent. This agent can deeply reason and analyze based on the current video memory and the question determine whether it can strictly and objectively provide confident answer using the existing memory. If an answer can be provided, it will output the result; if not, it will generate an unconfident flag. In addition to answering questions, this agent is also responsible for deciding whether to further explore the memory. The prompt design for it is shown in Tab. 7.
A.2 EgoMem Benchmark Details
Manual Annotation Process
Build a sequence of events that occur throughout the day. We require annotators to carefully review the daily videos in Egol ife, meticulously record the names of events that happen each day, and label each event with start and end times accurate to the minute, along with specific descriptions. Subsequently, arrange the entire day's events in chronological order to form complete event sequence, ensuring that the timeline is continuous and logically consistent.
Then, the question question-answer answer pairs for each task are annotated by referring to these event sequences.
For event understanding task, we adopt the following two steps annotate question- answer pairs.
1) Event subsequence extraction. A continuous target time period is selected from the complete event sequence of day, and then 4 to 6 events are randomly extracted from all events corresponding to that period. worth noting that the continuity requirement for event subsequences needs to be adjusted according to question types: some question types (such time period judgment and mask
24

---

There is currendy video with total duration of /video length) seconds. The following gives a general description what shown in the video during certain time periods: [Current Memory List (Time Period + Description)
Now, a question has been raised regarding the content descriptions of this vidco. (Question]
Please read the given video content descriptions and the question in depth, and determine whether you can accurately answer the given question solely based on the currently provided descriptions. If you can answer it with absolute confidence, please answer this question and provide the time periods of the video content you are referring to. The answer you provide must have completely and absolutely objective support in the video descriptions. Do not make inferences arbitrarily.
you think the current content descriptions of the video are still insufficient to accurately answer the question, please do not answer it and give me your reason.
Please output in strictly standardized dictionary format containing four key-value pairs:
"Confidence": A boolean value. Set it to True if you certain about the answer, and False if not.
"Answer": A string. This string must be enclosed in double quotes. When "Confidence" is True, fill in the answer content; when "Confidence" is False, fill in "No Answer".
Time Period": A list. When "Confidence" is True. fill in the list with time periods corresponding to the answer, each in the format of tuple (start time, end time); when "Confidence" is False, fill in "No Time". "Reason": A String. This string must be enclosed in double quotes. Show your reasoning about your judgment. You need ensure and check that your reasoning must be able absolutely support your answer.
Table 7: The designed prompts for the answering agent.
prediction) require events to be continuous, while question types such as event sorting and time point correspondence allow events to be discontinuous.
2) Designing six types of question-and-answer pair. Event Order: introduce interfering events from other time periods, mix them with sub-sequence events, and then ask for chronological ordering within the time range specified in the question stem. Temporal Judge: omit the time information of event sub- sequences and require providing the actual occurting time period. Temporal Align: separate events from their time periods, mix in modified incorrect times or times from other events as distractors. and require matching each event with the correct time period. Event Context: based on 2-4 consecutive events, require inferring their preceding or following contexts. Event Rectification: modify the original sequence and require identifying and correcting erroneous events. Event Reconstruction: require completing the obscured events in a continuous sequence.
By having annotators carefully carry out the above steps for full-day video, we can obtain six question-anddanswwe pairs focusing on different aspects of event understanding.
For detail perception task, the annotation process is simpler. We manually identify instantaneous visual features that appear in an extremely short timeframe from the already labeled event sequences, then create questions targeting these features. Highly confusing distractors are artificially designed, and each question stem is composed of background information plus a specific query. Each full-day video annotation includes six question-and-ansee pairs for detail perception.
Detailed Annotation Content
Taking the video from Day 1 of A6 SHURE in the benchmark an example, we will demonstrate the detailed content of our manual annotation in this subsection.
First, we carefully reviewed the full-day video, then annotated the event names, start and end times, and detailed content of the events that occurred on this day, as shown in Tab. 8.
Then, based on these events, we manually annotated six question question-and-aadd answer pairs for event under- standing (Tab. 9) and six question-ane andd--nd---nn ansswer pairs for detail perception (Tab. 10).
We can see that the QA pairs we have annotated are of high quality, effectively measuring the model's ability to understand cross-temporal relationships in long videos and capture details.
A.3 Ablation Study Details
For the Needle-in-A-Videe respectively. The specifie video sources and their lengths are described as follows. LVBench: EwskdNETNx8 (3600s), JlrzSvCsIjE (2791s), NeCO0G8AGLU (2415s), 014bbpvy2x0 (2005s),
experiment, the 10 long videos are from LVBench and MLVU
25

---

Equipment Assembly [11:09::5-11:23:499]
SHURE collaborated with other members in living room to complete the assembly electronic equipment. The man in a blue short-sleeved shirt distributed equipment to other members and provided on-site guidance.
Activity Planning Whiteboard Meeting 11:23::--12:18:55
SHURE was responsible for writing content on the whiteboard while other members discussed beside him, finalizing invited guests, activity content, and timelines.
Flower and Greeting Card Handling The man in a blue short-sleeved shirt brought flowers and greeting cards, and the woman in a white top, the woman in blue long dress, and the woman in white short-sleeved shirt handled them together.
1::34:3--22:001111
Jigsaw Puzzle Assembly 112:18:55-13:26:233
A6 SHURE and other members gathered around a table to assemble a wooden circular jigsaw puzzle. Whiteboard Assembly 11::0:54-13:17:2899
A6 SHURE, the man in blue short-sleeved shirt, and the woman in white short-sleeved shirt assembled the whiteboard stand together and hung up the whitehoard.
Second-Floor Chat 01:17:28-13:26:233
A6 SHURE went to the second floor to chat with the man a blue short-sleeved shirt and the woman in white short-sleeved shirt. During this period, A6 SHURE made two cups of pour-over coffee. Lunch Takeout Ordering 113:26:23-14:18:344
SHURE and the man a blue thort-sleever shirt jointly operated mobile phone order takcout, while others discussed and wrote lunch choices on the whiteboard.
Supermarket Shopping 11::1:33918::3::90
A6 SHURE and other members purchased beverages, hot pot bases, and other ingredients in the supermarket.
Metro Ride [18:37:08-185 1::::47008:
SHURE navigated and walked to metro station, then took the metro.
Classroom Self-Study (20:02:32-20:52:44]
A6 SHURE walked to his girlfriend's classroom and studied with her there.
Dinner at Off-Ca Off-Campus Noodle Shop (20:52:44-22:09:455]
SHURE and his girlfriend departed from the classroom, walked off-campus noodle shop for dinner, and then walked back to the classroom.
Table The manually annotated event sequence for A6 SHURE Day in EgoMem.
DXD31xG2bVQ (3205s), YIQugR7KSKg (3996s). MLVU: I (401s), ego__1 ago_81 (1250s), movide 1ovide10115 101_51 (796s), order_99 92 (1579s). The durations of these 10 videos approximately conform to a uniform distribution from 400 seconds to 4000 seconds. In addition, we obtained short videos each with duration of 10 seconds from the Internet and manually annotated each of these short videos with question-answer pairs about detail perception to form the "needles", shown in Fig. 9.
def calculate_ennteeeeeee (ntteptyttorrottt)::.--
words = lexe.
Counter(words)
total words len(words)
entropy = 0
for count in word_ werrd_counts.vveee vauts::allue():
probability count total_words
entropy = probability math.logg/(robbbiiitt) return entropy
Table 11: The code of Shannon entropy calculation.
For the Information Richness and Relevance in Backtracking experiment, the calculation code for Shannon entropy is shown in Tab. 11, and the designed prompt for the relevance evaluator is in Tab. 12.
A.4 Additional Experiments
To verify the good adaptability of our VideoLucy framework to various MLLMS and LLMs we conducted an ablation experiment to explore the performance of the entire agent system when different MI.LMs and L.LMs are replaced. To reduce experimental costs, we randomly selected 200 question-answer pairs from LVBench to form the test set for this ablation experiment. The
26

---

experimental results are shown in Tab. 13. We can see that our VideoLucy demonstrates excellent udaptability to various MLLMs and LLMs. Combining different MLLMs and LLMs consistently achieves good performance, indicating that Videol Lucy is highly robust framework for long video understanding agent systems. In addition, more powerful MILMMs and L.MMs generally yield more superior performance. For instance, Qwen2.5-VL-72B demonstrates a significant performance advantage over its 7B counterpart, while DeepSeek-R also outperforms Qwen2.5-72B a relatively greater extent. This indicates that with the advancement of MLLM and LLM technologies, the performance of VideoLucy can be consistently enhanced.
MLLMs
LLMs
Acc.
Owen2.5-VL-7B
DeepSeek-RI
57.5
Qwen2.5-VL-72B
DeepSeek-RI
64.0
InternVL2.5-8BB
DeepSeek-RI
54.5
IntemVL22.57788
DeepSeek-R1
65.5
Qwen2.5-VL-7B
Qwen2.5-72B
48.0
Qwcn2.5-VLL-72B
Qwen2.5-72B
52.0
Intern VL2
Qwen2.5-72B
44.5
Intern VL2
Qwen2.5-72B
54.0
Table 13: Ablation study on the performance with different MLLMs and LLMs.
How old is Lucy? What is her specific date of birth?
VideoLucy
Answer:
Lucy is about 25 years old, but her specific date of hirth cannot be confidently inforred from existing video descriptions.
Reason:
In the time period (2891108 Os 2900.0s), news bmoeccastt clip eppears, in which Chinese subtitle translates to "The police are pursing an American woman aged about 25." inferved that Lucy apgroximately 25 years old. However, none the detailed descriptions explicitly state Lucy's specific date of birth, a confident answer cannot be given regarding it.
Therefore,
1:22.04
1-22.39
At the end of the movie, what is the professor wearing white coat holding in his hand?
VideoLucy
Answer:
The professor is bolding a black fountain pen his hand.
Reason:
the final scen of the movie, description of a time period (5290s 5300s) directly states that elderty man white
holding black fountain pen hand, so
confident answer can be given.
Figure 10: Failure Case Analyses. In Example since no clues related the specific date of birth are reflected in the coarse-graine memory, and the question has no scene prompts in this regard, VideoLucy has difficulty locating the relevant time period within five iterations. In Example due to the hallucination phenomenon occurring in the MLLM, which acts as the system's 's "eye". the object was not correctly understood and identified, leading VideoLucy to output an incorrect answer as well.
A.5 Failure Case Analyses
Like existing agent systems, the performance of our Videol Jucy also constrained certain extent by the inherent capabilities of the MLLMs and LLMs within the system. We demonstrate
27

---

two representative failure cases in Fig, 10. In the first case, during the initial extraction of the video's coarse-grained memory, detailed information-sutc as the passport details in the frame-was was overlooked, with only a general description ("a passport appears") retained. When users queried Lucy's exact date of birth without contextual cues, the L.LM's inherent limitations prevented from inferring that the date might be in the passport. Consequently, the system failed to locate the passport scene within five retrieval cycles, lacking any relevant information to answer the query accurately. In the second case, the user's explicit scene prompt ("the end of the movie") enabled the system to directly locate the target time period, followed by fine-grained information retrieval. However, due to MLLM hallucination, the model misidentified the black flash drive in the professor's 's hand black fountain pen-attributed attributed to their visual similarity. Then, the system's 's default trust in MLLM outputs led to erroneous response. Evidently, the system's response failures fundamentally stem from the current limitations of MILMs and LLMs. However, this can be significantly mitigated by integrating more capable models. Thus, advancing MILMM and LLM technologies is imperative for future improvements, which will directly enhance Videolucy's ' capabilities in tandem.
A.6 Broader Impacts
The development of long video understanding technology has brought profound changes to various fields of society. Its positive impacts are mainly reflected in education, healthcare, the content industry, scientific research and etc. In the field of education, this technology supports personalized learning. Students can ask real-time questions about teaching videos and receive answers. breaking through the limitations of time and space. In medical scenarios, doctors can quickly analyze the key information in surgical videos to assist in case review and teaching. In addition, users of video platforms can accurately retrieve content through natural language, significantly enhancing the consumption experience. Researchers can also use this technology to automatically analyze experimental videos, improving data processing efficiency.
However, the application of this technology has also given rise to series of negative impacts. Privacy and data security risks are the most pressing concerns, as the analysis of videos in private settings may lead to the leakage of personal sensitive information. The authenticity of information also faces challenges, as the combination of deepfake vidcos and this technology can easily lead to the spread of rumors. At the same time, the structure of the labor market has been impacted, resulting in decrease in the demand for jobs such as content moderation and media editing. Ethical and legal issues, including algorithmic bias and copyright disputes, also urgently need to be addressed.
To rationally utilize long-video understanding technology, multi-level governanee framework integrating technology, policy, and social coordination is needed. Technologically. develop privacy- preserving computing like federated learning and differential privacy to safeguard data during analysis. Implement multi-modal cross- tross-validation mechanism, verifying text, images, and audio, to boost content understanding accuracy and anti-forgery capabilities. Policy-wise, expedite legislation to define video data collection, storage, and usage boundaries. Establish an algorithm registration and transparency review system, mandating technology users to disclose core algorithm logic and data sources. Set up specialized regulators rigorously assess public-interest applications, such as security monitoring and medical video analysis. Socially, reinforce technological ethics education, heighten public awareness of tech limitations, and encourage enterprises, academia, and social groups to co-create industry self-regulatory guidelines. These actions will foster technology development to benefit society while respecting privacy and ensuring fairness.
A.7 Limitation
Although our VideoLucy, benefiting from the hierarchical memory structure and iterative backtracking mechanism, can effectively understand long videos better than existing methods. However, like almost all existing agent-based systems, its performance is still affected by the respective capabilities of each agent in the system.
The accuracy of video MLLMs, which act as the system's 's "eyes", in understanding short videos will directly affect VideoLucy's subsequent backtracking process. Our VideoLucy utilizes existing video MI.MMs to describe given video clips and extract corresponding memories. Obviously, if an MILM with limited capabilities fails to effectively understand given video and provides incorrect descriptions (i.c., hallucinations occur), this will negatively impact the factual consistency of the
28

---

memories stored in VudeoLucy, thereby becoming unavoidable interfering information in the system. However, improving the capabilities the captioner can theoretically significantly enhance system performance. As wransformation-fiiene Videoluccy. That is, with the development of MI.LM technology, the performance of our Videol.cy will also achieve gradual improvement.
framework, any MIL.Ms can be easily integrated into our
As the core of the entire system's reasoning, the LLM's capabilities in text understanding and instruction following will also affect the system's 's performance. Our VideoLucy leverages existing LLMs to understand memories (textual descriptions) and requires them to complete corresponding tasks, such as question answering, key time positioning, missing information analysis, etc. Similarly, if an LLM with limited capabilities is unable fulfill the above responsibilities and performs erroneous analysis and processing of the text, it will affect the system's 's performance to a certain extent. Additionally, the maximum video length that VideoLucy can process is also affected by the LLM's maximum context limit, which serves as a bottleneck for super-long videos longer than existing benchmarks, such as hundreds of hours. However, like other agent systems, our Videolucy training- free framework, allowing any LLM with improved capabilities to be seamlessly integrated into it. Therefore, with the development of LLM technology, both the performance of our VideoLucy and the maximum video length it can process will improve steadily.
Additionally, like existing video agent systems, since our VideoLucy requires processing using LLMs. it generally takes relatively more time to derive confident answers compared to end-to-end video MLLMs. However, considering that the memories obtained in our framework can be stored, for multiple different questions about the same video, we can fully leverage the video's s pre-existing memories to achieve faster responses, instead of repeatedly performing inference on the video's s information existing video MILMs do. This can reduce the time overhead of reasoning to a certain extent, which is conducive to deployment in practical applications.
analytical
In future work, we will focus on exploring how to reduce the impact of unavoidable noise information in agent systems and how improve the operational efficiency of agent systems through better framework design while ensuring performance. We believe that our VideoLucy, as an agent system with comprehensive dynamic memory of long videos, can bring new inspirations to this community.
A.8 Inference Time
We acknowledge the importance of inference efficiency in practical applications and provide an analysis of VideoLucy's 's inference time compared other agent-based methods.
The inference overhead VideoLucy primarily stems from two components: the MLLM processing short video clips and the LLM reasoning over long textual contexts. Both components can be effectively accelerated using modern inference frameworks. In our implementation, we deployed Dwen2.5-VL-7B locally on A100 GPUs using vLLM, achieving efficient batch processing that generates textual descriptions for one-hour videos within dozens of seconds. For the LLM component, utilized the official API of DeepSeek-R. due to the high cost of local deployment.
To ensure fair comparisons across different agent-based systems, we adopt the number of LLM calls as our primary efficiency metric. This approach effectively eliminates confounding factors from hardware configurations and deployment environments, providing an objective basis for comparing methodological efficiency.
We conducted controlled experiment using 42 question -answrr pairs randomly selected from the EgoMem dataset, with the maximum number of LLM calls limited to 20. The results demonstrate that VideoLucy requires significantly fewer LLM calls (6.3 on average) compared VideoAgent (14.1) and VideoTTree (9.6), indicating substantial advantage in inference efficiency. This efficiency gain stems from our optimized reasoning framework that minimizes unnecessary iterative processes while maintaining high answer quality.
29

---

[Event Order]
A6 SHURF forgot the events he experienced between 11 11:00 noon and 1:30 pm on DAY1 (a. Device assembly;
b. Activity planning whiteboard meeting; c. Jigsaw puzzle; d. Supermarket procurement). If you were to sort the events that occurred during this period, what would be the correct order?
A. (a)(b(c) B.(a)c)(b) C. (b)(a)() D. (c)()(()
Evidence: Device assembly (11:09:45-11:223449 - Activity planning whiteboard meeting (11:23:49-
12:18:55) + Jigsaw puzzle
1::2633..3. Supermarket procurement occurred from 17:18:39 to 18:37:08, not within 11:00-13:30, so options with are excluded. The order in option A fits the time logic.
[Temporal Judgel
When summarizing his work, A6 SHURE wrote the sequence: [Device assembly -> Activity planning whiteboard meeting -> Jigsaw puzzle]. What time range should A6 SHURE annotate?
A. DAY1: 11:00 12:00
B. DAY1: 11:00 13:00
C. DAY1: 11:00 13:30
D. DAYI: 11:00 14:00
Evidence: Device assembly (11:09:45-1) 11:23 :::4:49) + Activity planning whiteboard meeting (11:23:49-
12:18:55) - Jigsaw puzzle
13::266333. Total span: 11:09::4-11:26:22 matching C (11:00-13:30),
[Temporal Align]
SHURF only recorded what he did, but did not record the start and end times cach thing. Now there are some time periods for A6 SHURE to choose from. Event sequence: [Jigsaw puzzle + Whitchoard assembly - Second-floor chat]. Time options: 12:00-13:00 b. 12:15-13:30 c. 12:50- 13:20 d. 13:15-13:30
13:00-13:30 f. 12:18-13:26 Can you help A6 SHURE find the correct time period for each event?
A. 1-ff/--c// /3-rd
B. 1-ta/1/22222 2 3->e C. 1--b/ 2-> /3
D. 1-+f 2->e/>>e// 3-
Evidence: Jigsaw puzzle (12:18:55-)) 13::6233,, corresponding to option f; Whiteboard assembly (12:50:54-
13:17:28), corresponding to option c; Second- chat (13:17:28. 12222881:::::::: 13:26:23), corresponding to option d. The
correspondence in option A is correct.
[Event Context]
After A6 SHURE completed the jigsaw puzzle, what was the next event that happened?
A. Whitehoard assembly B. Second-floor chat C. Takeout lunch ordering D. Supermarket procurement
Evidence: According to the time sequence, after the jigsaw puzzle 11:::1551::2::220 is completed, the next takeout lunch ordering
1:26:23-14:18:3: The whitehoard assembly 12:500:54-13:7:::: was carried out during the jigsaw puzzle process, the second-floor chat (13:17:28- (13.17:28-13:26223) also occurred during the jigsaw puzzle and before the takeout lunch ordering. and the supermarket procurement 17:11839-18:37:00 an event that happens much later. Therefore, the next immediate event is the takeout lunch ordering.
[Event Rectification
A6 SHURE described series of consecutive actions as: [Whitehoard assembly + Event planning whiteboard meeting + Second-flow -fooo chat], but her statement is incorrect. What is the correct sequence and the error analysis?
A. Correct: Event planning whiteboard meeting- - Whiteboard assembly + Second-floor chat. Ertor reason:
Incorrect order (event planning whitehoard meeting should precede whiteboard assembly)
B. Correct: Whiteboard assembly - Second- floor chat + Event planning whiteboard meeting. Error reason:
Disordered order
C. Correct: Event planning whiteboard meeting + Second-floo chat + Whiteboard assembly. Error reason: Incorrect order
D. Correct: Second-floor chat Event planning whitchoard meeting Whiteboard assembly. Error reason: Completely reversed core event order
Evidence: The correct sequence should be Event planning whiteboard meeting (11:23:49-12:138 + Whiteboard assembly (12:50:54-13:
- Second-floor chat (13:17:2--13:2622) A6 SHURE's descrip-
tion reversed the order of "Event planning whiteboard meeting" and "Whiteboard assembly". Option A corrects the core order."
[Event Reconstruction] In A6 SHURE's diary, the record of the order of her participating activities is unclear: [Activity planning whiteboard meeting] + (?) - [Whiteboard assembly] -> (?) - [Supermarket procurement]. Which of the following events be used to fill in the unclear parts?
A. Jigsaw puzzle/ Second-floor chat
Second-floor chat/ Takeout lunch ordering D. Takeout lunch ordering / Second-fioo chat
C. Jigsaw puzzle/ / Takeout lunch ordering
Evidence: Activity planning whiteboard meeting (11:23:49-12:18:55
+ Jigsaw puzzle (12: (12:18:55-13:2)) + Whiteboard assembly 13:10:54-13:17::88 Takeout lunch ordering 113:26:33-14:18::: ket procurement (17: immediately after the activity planning whiteboard meeting. The second mask should filled with "Takeout lunch ordering", which is after the whitchoard assembly and hefore the supermarket procurement. Option C correctly matches the chronological order.
Supermar-
The first mask should be filled with "Jigsaw puzzle", which starts
Table 9: The manually annotated six event understanding question-answer pairs for A6 SHURE Day in EgoMem. The correct options are in blue. Each pair is also annotated with corresponding evidence.
30

---

[Detail Perception 11
When SHURE attended the whiteboard meeting on Day 1, there were three categories of people written on the whiteboard. The first was the invited guests, the second was the invited neighbors. What was the third category?
A. Teachers B. Crew members
Classmates D. Colleagues
Evidence: On Day from 11:26:57 to 11:33:15, SHURE wrote three categories people on the whitehoard. The first was the invited guests, the second was the invited neighbors, and the third was the crew members.
[Detail Perception 2]
Before SHURE assembled the whiteboard on Day 1, who used scissors to open the express box containing whiteboard stand?
A. The woman black top B. The boy in blue short-slecved shirt
C, SHURE D. The girl in white short-sleeved shirt
Evidence: On Day I, from 12:50:27 to 12:54:46, SHURE used scissors to open the express box containing the whiteboard stand.
[Detail Perception 3]
On Day 1, during the car ride to the supermarket, who sat next to SHURE in the car?
A. The woman in black top B. The boy in blue short-sleeved shirt D. The girl in white short-sleeved shirt
C. The woman blue dress
Evidence: Day I, from 17:11:15 to 17:13:04, during the car ride to the supermarket, the boy in blue
short-sleeved shirt sat next to SHURE.
[Detail Perception 4]
On Day 1, after SHURE finished shopping with the team and took the subway, what was the name of the subway station they walked to?
A. Jiukeshu Station
B. Guoyuan Station
Caodi Station D. Senlin Station
Evidence: On Day 1, from 18:37:23 to 18:45:22, SHURE walked to Jiukeshu Subway Station.
[Detail Perception 51
On Day when SHURE went to the classroom find his girlfriend. which row of the classroom was their
seat in?
A. The first row
B. The second row
C. The third row
D. The fourth row
Evidence: On Day 1, from 20:05:35 to 20:07:10, SHURE entered the classroom, found his girlfriend in the
fourth row and took a seat.
[Detail Perception 6] evening of Day when SHURE went have dinner with his girlfriend, what was the number on waiting sign placed on the table while waiting for the food after ordering?
A. 3
B. 4 C. 43
D. 34
Evidenee: On Day 1, from 21:13:22 to 21:20:12, 12, the number on the waiting sign the table while waiting for the food after ordering dinner was 43.
Table 10: The manually annotated six detail perception question-answer pairs for A6 SHURE Day in EgoMem. The correct options are in blue. Each pair is also annotated with corresponding evidence.
Please complete the following task: Carefully analyze the given Text and Question.
Determine whether the Text contains descriptions information relevant the Question. Score the relevance on a scale of to based on the following criteria: point: The Text has no relevance to the Question, and there is no content related to the Question in the Text at all.
points: The Text has very weak relevance the Question, with only minimal amount of unrelated indirect descriptions.
The Text has some relevance to the Question, containing some relevant information, but is not comprehensive or in depth enough.
points:
4 points: The Text has relatively strong relevance to the Question, containing a substantial amount of relevant information and being able respond the Question fairly well.
points: The Text highly relevant the Question, fully and thoroughly covering all the key information required by the Question.
Output the final score in the format of "Scoring result: X points", where X is an integer between and 5.
Given Text: (Text)
Given Question: (Question)
Table 12: The designed prompts for the relevance evaluator.
31

---

short scene appears in the video where an animal doll performs on magnificent stage.
1. What kind animal doll the main performer this scene? Ciraffe B. Tiger C. Gorilla D. Zebra
2. What musical Instrument being played? A. Violin
B. Guitar
Plano
D. Saxophone
3. What color of clothes is it wearing?
A. A dark brown leather coat and green inner garment.
B.A blue leather coat and green inner garment.
C.A dark brown leather coat and a blue inner garment.
D. A blue leather coat and blue inner garment.
4. What is written the plaque hanging above the stage? A. sing tour B, animal show. C. dancing show. D. song and piano.
A short scene of a basketball game appears in the video.
1. What is the jersey number of the play er with the ball on offense? A.
B. 16
C. 23 D.
2. What color of sneakers is the player with the ball on offense wearing? A. Pink
B. Blue
C. Black
D. White
3. Wha kind of medal are the four players in the photo holding?
D. They are not holding any medal 4. hat brand of sports jacket are the four players in the photo wearing?
A. Bronze medal
B. Silver medal
C. Gold medal
A. Nike
B. Adidas
C. Puma
D. Under mour
A short scene appears in the video, in which man dressed in ancient costume is sitting on the ground and looking into mimor
1. When the man looks at his feet, how many moles are there on his feet unexpectedly? A. 1 B.
C.
D.4
2. What style of shoes is the man w vearing?
A. One foot is bare and the other foot is wearing a straw sandal.
B. Wearing pair of casual sports shoes.
C. One fool is bare and the other foot is wearing a sports shoe.
D. Both feet are not wearing any shoes.
3. W hat style of pants is the man wearing?
A. Dark gray long pants decorated with red flowers.
B. Dark gray long pants decorated with orange flowers.
C. White long pants decorated with red flowers.
D. Pure gray long pants.
4. What does the man see in the mirror? A. The Monkey King
B. Zhu Bajie
C. Sha Heshang
D. Tang Seng.
short live news report. about a snooker match appeers in the video.
1. Who is reported to have won the championship in this scene?
B. Ding Junhui 2. What is the time of the news broadcast in this scene? A. May 6th
A. Zhao X Xinlong
C. Ronnie O'S O'Sullivan
J.Judd Trump
B. May 7th
C. May 8th
D. May 9th
3. hat color of ball is the snooker player hitting in this scene? A. Black ball
B. Pink ball C. Blue ball
D. Green ball
4. hat color of clothes is the winning player wearing in this sceme?
A white long-sleeve sleevee shirt with black eevelless waistzoat.
B. A black long-sleevedd shirt with a white sleeveless waistzoat. Just a black long- sleeved shirt.
C. just white long-sleen sleeved shirt.
A scene appears in the video, in which elderly man standing beside truck is having conversation with shirtless man.
1. What does the old man ask the young man to do?
A. Ask the young man to lend him money. A ak the young man help collect the rent.
C. Ask the young man to pay the rent.
D. Pay the rent for the young man.
2. What color of hat is the young man wearing? A. Black
B. Gray
C. Nol wearing a hat
D. Green
3. What are the names of the two of them?
C. Roccyy Jack 4. W hat does the shirtless young man do after the conversation?
A. Brian aason Jason
B. Bob and Jack
D. Trump and Bob
A. He opens the door and walks into the room.
B. He goes down the stairs.
C. He drives away.
D. He jumps off the balcony.
Figure 9: Display of five short videos used as "needles". Each short video contains four manually annotated question-answer pairs about detail perception, with the correct options marked in green.
32


Reviews:
[{"review_id": "4HHL0Nf74t", "reviewer": "Reviewer_RcwM", "summary": "This paper introduces VideoLucy, a deep memory backtracking framework for long video understanding that leverages a hierarchical memory structure and agent-based iterative backtracking to effectively capture temporal context and preserve fine details. By progressively mining question-relevant information, VideoLucy overcomes the limitations of sparse frame sampling and improves reasoning over long video sequences. Extensive experiments and a newly proposed EgoMem benchmark demonstrate that VideoLucy outperforms state-of-the-art and even proprietary models like GPT-4o on several long video understanding tasks.", "questions": "**A. About the Writing**\n\n**A.1** Please consider revising the introduction's first paragraph, especially the section on video understanding. For readers unfamiliar with the movie \"Lucy,\" it is difficult to grasp the context of the paper.\n\n**A.2** The concept of the \"human recollection process from coarse to fine\" is mentioned but not explained in either the abstract or the introduction. Providing a clear explanation or an example would help the reader better understand this idea.\n\n**B. About the Methodology**\n\n**B.1** The method divides videos into equally sized clips, which may cause important events to be split across multiple clips. Have you considered alternative segmentation methods or analyzed the potential drawbacks of uniform segmentation?\n\n**C. About the Experiment**\n\n**C.1** What are the runtime and memory requirements of your method? Including a chart or table to illustrate these aspects would be helpful.\n\n**C.2** Why do the baselines differ in Tables 1, 2, and 3? Is this due to certain baselines being incompatible with some datasets, or were they specifically selected by the authors?\n\n**C.3** It is unclear which models are used in CapAGT, LocAGT, and other variants. It would be better to specify the base models used and compare your approach with their baseline capabilities, ideally through experiments using different underlying models.\n\n**C.4** The evaluation on EgoMem shows that your method has a significant advantage in temporal understanding. Can you clarify which aspects of your approach contribute most to this improvement?\n\n**C.5** Please analyze the improvements achieved on different tasks. Since the proposed benchmarks in the paper are primarily focused on temporal understanding, is the performance gain on benchmarks such as Video-MME and LVBench mainly attributable to enhanced temporal sequence understanding?", "limitations": "The authors adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\nThe idea presented in this paper is novel and holds significant potential for practical applications in video tasks. The paper is clearly written and is supported by comprehensive experiments, which leaves a strong impression. The structure is well-organized, and the illustrations are of high quality.\n\n**Weaknesses**\n\nWhile the paper presents promising results, it lacks sufficient clarity in its introduction and explanation of key concepts, which may hinder understanding for readers unfamiliar with certain references. Additionally, the methodology’s reliance on uniform video segmentation may introduce limitations, and important details regarding experimental baselines, model choices, and resource requirements are insufficiently discussed. A more comprehensive analysis of these aspects would strengthen the work.", "was_revised": true, "final_justification": "I have reviewed the authors' rebuttal and the revised manuscript. They have successfully addressed all the key concerns raised in my initial review, and the manuscript is now significantly improved.", "rebuttal": "We sincerely thank Reviewer `RcwM` for the positive and constructive review. We are grateful for your recognition of our idea, strong experimental results, and clear manuscript presentation. Below, we address your insightful comments and suggestions in detail.\n\n---\n> **`A.1`:** *\"Please consider revising the introduction's first paragraph, especially the section on video understanding. For readers unfamiliar with the movie ‘Lucy,’ it is difficult to grasp the context.\"*\n\n**A:** Thank you for the suggestion. We have revised the first paragraph to provide clearer context for long video understanding before introducing the analogy to the movie Lucy. The revised paragraph now reads:\n\n*\"Long video understanding is a highly concerned task, with the core objective of accurately and objectively answering various user questions based on the entire video content. This process demands that a system possess a comprehensive memory and grasp of almost all details within the video; otherwise, information gaps could lead to inaccurate answers.*\n\n*This need for comprehensive memory brings to mind a scene from the movie Lucy. The protagonist, Lucy, gains full access to her brain's potential due to an accident, acquiring an exceptionally powerful memory. She can recall every detail of her life from birth, even the sensation of her mother stroking her forehead during infancy. This extraordinary ability to trace back and precisely capture all information, whether instantaneous frames or continuous events, is exactly the goal we pursue in the long video understanding task.\"*\n\nWe have incorporated this revision into the revised manuscript.\n\n---\n> **`A.2`:** *\"The concept of the 'human recollection process from coarse to fine' is mentioned but not explained in either the abstract or the introduction.\"*\n\n**A:** We agree that a clearer explanation of this concept would help readers better understand our method's motivation. Drawing from cognitive science, human recollection typically proceeds from coarse to fine, **starting with general impressions and gradually retrieving finer details.** For example, recalling a birthday party might begin with the memory of a lively room, and then be refined into specifics like a gift exchange or a candle count.\n\nVideoLucy is inspired by this process. It uses a hierarchical memory structure that separates coarse and fine memories across time scales, coupled with agent-based iterative backtracking to simulate how humans actively retrieve and refine information. We have added this explanation and an illustrative example in the revised introduction.\n\n---\n> **`B.1`:** *\"Have you considered alternative segmentation methods or analyzed the potential drawbacks of uniform segmentation?\"*\n\n**A:** This is a valuable point. We **initially explored overlapping segmentation** (e.g., the first clip covers 0-100 seconds, the second 90-190 seconds) to reduce event truncation. This aimed to mitigate direct event truncation by allowing event endings in one clip to serve as beginnings in the next.\n\nHowever, experiments showed this design **did not improve understanding accuracy and, conversely, increased the number of memories**, negatively impacting system efficiency. Thus, we ultimately abandoned overlapping segmentation.\n\nOur qualitative experiments revealed why the overlapping design failed: **LLMs possess the ability to integrate and understand continuous events.** For example, if an event spans two clips (e.g., 95-105 seconds across 0-100 and 100-200 second clips), during the memory search process, the LLM can still roughly locate that the event should occur at the end of the first clip and the beginning of the second clip. It can then provide a refined search range that connects these two clips, effectively mitigating the potential consequences of direct truncation to a certain extent. \n\nWe sincerely appreciate your suggestion, which helps us improve the explanation of our method.\n\n---\n> **`C.1`:** *\"Runtime and memory requirements.\"*\n\n**A:** VideoLucy is a training-free agent. Its resource consumption comes mainly from its MLLM and LLM components.\n\n- **Memory and runtime requirements:** Local MLLM memory usage (Qwen2.5-VL-7B) scales with video frame count and concurrent clips. An 80GB A100 GPU can handle 10 concurrent 200-frame clips in our tests. More memory allows faster, parallel inference. Limited memory necessitates processing fewer clips simultaneously, extending total inference time. For the LLM (DeepSeek-R1), we used its official API due to substantial local deployment requirements. This meant runtime consumption varied with API response times, preventing precise local performance measurement.\n- **Runtime performance:** Our method's runtime is affected by video length, question difficulty, and API response times. To quantify this, we processed 20 questions for each of four videos (400s, 1600s, 2800s, 4000s) from the Needle-in-A-Video-Haystack dataset. The average times are presented below:\n\n|Duration|Local MLLM Time (s)|LLM API Time (s)|Total (s)|\n|-:|:-:|:-:|:-:|\n|400s|7.8|36.8|44.6|\n|1600s|12.4|42.9|55.3|\n|2800s|14.7|60.8|75.5|\n|4000s|18.9|72.9|91.8|\n\nAs anticipated, runtime scales with video duration, primarily due to LLM API call time. Using a faster, more stable LLM API will accelerate inference. We have included this analysis in the revised version to clearly illustrate these aspects.\n\n---\n> **`C.2`:** *\"Why do the baselines differ in Tables 1, 2, 3?\"*\n\n**A:** Thanks for raising this concern. The differences in the baseline methods among these tables are mainly due to the fact that many video understanding methods do not have officially published test results on all three benchmarks.\n\nAs stated in the captions of each table, the results consist of two parts:\n- The top-performing results on each leaderboard as of May 15, 2025 (the deadline for paper submission);\n- Results from recent, representative papers that, while not yet officially on the leaderboards, offer valuable insights into the latest advancements.\n \nBecause resource investments and testing plans vary, many methods have only published results on one or two datasets. This naturally causes discrepancies in the baseline methods available for comparison across the three tables, not due to any deliberate selection on our part.\n\nWe hope the above response addresses your concerns.\n\n---\n> **`C.3`:** *\"It is unclear which models are used in CapAGT, LocAGT, and other variants. It would be better to specify the base models used and compare with their baseline capabilities.\"*\n\n**A:** We appreciate your feedback about the clarity of the base models and your suggestion for further comparisons. All agents in our system consistently use `Qwen2.5-VL-7B` as the MLLM and `DeepSeek-R1` as the LLM (please refer to Line 240).\n\nAs you suggested, we have now included a direct performance comparison between VideoLucy and the original `Qwen2.5-VL-7B`, using results from its official technical report:\n\n|Model|VideoMME|MLVU|LVBench|\n|-|:-:|:-:|:-:|\n|Qwen2.5-VL-7B|65.1|70.2|45.3|\n|VideoLucy|72.5|76.1|58.8|\n\nAdditionally, within the limited rebuttal time, we have supplemented comparative experiments using different underlying models. The experimental setup is consistent with Table `9` in the supplementary material. Results are as below:\n\n|Model|Acc. (original)|Acc. (+VideoLucy)|\n|-|:-:|:-:|\n|Qwen2.5-VL-7B|42.5|57.5|\n|Qwen2.5-VL-72B|45.5|64.0|\n|InternVL2.5-8B|40.5|54.5|\n|InternVL2.5-78B |42.0|65.5|\n\nIt can be seen that VideoLucy shows performance advantages with each baseline model. We have added these results to the revised supplementary material as you suggested.\n\n---\n> **`C.4`:** *\"Can you clarify which aspects of your approach contribute most to the improvement?\"*\n\n**A:** Thanks for your insightful comment. VideoLucy's significant improvement in temporal understanding primarily comes from two key features:\n- **Hierarchical memory structure:** This structure uses both coarse-grained memory (for broad event-level dynamics) and fine-grained memory (for detailed temporal changes within events). This multi-scale approach helps the model understand both overall event progression and subtle temporal variations.\n- **Agent-based iterative backtracking:** This mechanism integrates memory across different time segments, allowing the model to establish cross-event temporal connections. This is crucial for understanding how events unfold and interact, especially for long-term temporal dependencies in EgoMem.\n\nTogether, these two aspects enable VideoLucy to effectively capture temporal dependencies and integrate information over extended periods, leading to its superior performance on EgoMem's temporal understanding tasks.\n\n---\n> **`C.5:`** *\"Please analyze the improvements achieved on different tasks.\"*\n\n**A:** Thank you for your valuable feedback. We evaluated VideoLucy on three benchmarks, focusing on different aspects of long video understanding. For detailed analysis, we use LVBench as an example due to its explicit task categorization.\n\nLVBench includes six task categories: Temporal Grounding (TG), Summarization (Sum), Reasoning (Rea), Entity Recognition (ER), Event Understanding (EU), and Key Information Retrieval (KIR).\n\nVideoLucy significantly improved performance on EU (9.1%), KIR (13.4%), and TG (6.2%). This highlights our framework's strength in **capturing continuous event dynamics and preserving critical details** through its hierarchical memory and iterative backtracking. VideoLucy also showed smaller yet still significant gains on ER (0.9%), Rea (1.2%), and Sum (0.3%), further demonstrating its overall effectiveness in video understanding.\n\nTask-specific results confirm that performance gains on various benchmarks are mainly due to improved temporal understanding and detail preservation. We hope this addresses your concerns.\n\n---\nWe would like to sincerely thank Reviewer `RcwM` again for the valuable time and constructive feedback provided during this review."}, {"review_id": "BxUnELb8ag", "reviewer": "Reviewer_Dmx9", "summary": "This paper proposes an agent-based long video understanding method VideoLucy, which imitates the human memory recall mechanism from coarse to fine. In order to verify the effectiveness of the proposed method, this paper also conducts comprehensive evaluation on three commonly used long video understanding datasets such as MLVU, VideoMME and LVbench and the newly collected EgoMem Benchmark. The results on all benchmarks show the effectiveness of VideoLucy.", "questions": "In general, I think the research motivation and method design of this paper are reasonable, but there are some aspects that can be improved, such as: 1) the description of the method details can be clearer (weaknesses 1 and 2); 2) the type of questions set for the benchmark can be more diverse (weakness 3 and 4); 3) the performance comparison can be more comprehensive (weakness 5). In addition, the writing of this paper can be further polished (such as the expression of Lines 128-146).", "limitations": "No negative social impact has been found so far.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. This paper imitates the human memory mechanism to design a video understanding agent from coarse to fine, and the research motivation is reasonable.\n2. The performance advantage of the proposed method is significant, and the long video understanding ability exceeds the advanced models in the industry such as GPT-4o.\n3. The long video understanding ability verification experiments are comprehensive.\n\nWeaknesses:\n1. Is there a problem with the $y_i = k$ condition in the Line 132? In this way, only the first K video frames can be sampled, and each frame is a clip?\n2. The method in this paper needs to design different hyperparameters for different datasets. Can some general design references be given?\n3. Among the 6 types of event understanding tasks in the EgoMem benchmark, Event Order, Event Rectification, and Event Reconstruction essentially test the same ability of the model. What is the basis for this split?\n4. Although the problems on the EgoMem benchmark involve the understanding of multiple events, the events are relatively independent of each other and lack interactive relationships. Have the authors considered a type of question that requires comprehensive review of information from multiple events to get an answer, such as: What do the people in event 1 and event 2 accomplish together?\n5. GPT-4o's ability to understand long videos is not its strength. If the proposed method can be compared with a model such as Gemini2.5-pro that has optimized long video understanding ability, it will better illustrate the effectiveness of this method.", "was_revised": true, "final_justification": "After receiving the author's response during the rebuttal phase, my confusion has been resolved. Furthermore, I have read the comments of the other reviewers. Overall, I believe the strengths of this paper outweigh its weaknesses, so I will maintain my original score.", "rebuttal": "We sincerely thank Reviewer `Dmx9` for the thoughtful feedback and recognition of our work’s motivation, contributions, and empirical results. Below, we address each concern in detail.\n\n---\n> **`Q1`:** *\"Is there a problem with the $y_i = k$ condition in the Line 132? In this way, only the first K video frames can be sampled, and each frame is a clip?\"*\n\n**A:** Thank you for raising this question. It is likely that our unclear expression has led to a possible misunderstanding. The process of dividing the clips is as follows:\n\nFor a video with $N$ frames, we use $f_i^{y_i}$ to represent each frame, where $i$ is the index of the frame in the video, and $y_i$ is the label indicating which clip the frame belongs to. Then we divide the video into $K$ short clips, and we use $v_k$ to represent each short clip, where $k$ is the index of the short clip. The meaning of $f_i \\in v_k$ if $y_i=k$ is that if the clip label $y_i$ of $f_i$ is $k$, then this frame belongs to the $k$-th segment.\n\nIn this way, we can control the number of frames in each short clip by simply setting different values of $K$, that is, controlling the temporal scope of memory. When $K=1$, no division is performed, and the memory degrades into a rough understanding of the entire video. When $K=N$, the division is done frame by frame, with each frame being a short clip.\n\nFollowing your suggestion, we have polished the expression of this part in the revised version to avoid possible misunderstandings.\n\n---\n> **`Q2`:** *\"The method in this paper needs to design different hyperparameters for different datasets. Can some general design references be given?\"*\n\n**A:** Thank you for your concern regarding the hyperparameters for different datasets.\n\nTuning temporal scopes (i.e., the time window sizes for partitioning coarse/fine-grained memories) across datasets is necessary due to **significant variations in video durations;** a fixed scope would be impractical.\n\nPreliminary tests on videos of varying lengths showed that too few coarse-grained memories restrict initial information for iterative backtracking, increasing search iterations. Conversely, excessive coarse-grained memories introduce redundancy, unnecessarily lengthen context, and hinder retrieval efficiency.\n\nTo address this, we derived general design principles through empirical comparisons across multiple datasets:\n- For **coarse-grained memories:** The temporal scope should be set such that the number of coarse-grained memories falls within a reasonable range (typically 10–36). This range is chosen based on common video durations (from minutes to hours in most datasets) to balance initial information richness and context efficiency. \n- For **finer-grained memories:** The temporal scope is approximately one-tenth of the level above. This serves as a starting point and can be adjusted based on the dataset's sensitivity to detailed information.\n\nThese principles balance the trade-off between initial information richness and context length efficiency while preserving the integrity of the hierarchical memory structure. They also provide a general reference for initial hyperparameter tuning on new datasets, reducing trial-and-error efforts.\n\n---\n> **`Q3`:** *\"Among the 6 types of event understanding tasks in the EgoMem benchmark, Event Order, Event Rectification, and Event Reconstruction essentially test the same ability of the model. What is the basis for this split?\"*\n\n**A:** Thank you for your valuable comments. While the three tasks do share a foundation in assessing models' ability to understand temporal event sequences, their core objectives and required capabilities differ.\n\n- The **Event Order task** centers on accurately selecting events within a target time window from a set containing distractors, then establishing their temporal sequence. This demands not only recognition of events' temporal attributes but also the ability to identify and exclude irrelevant information, ultimately posing a dual challenge of \"screening and ordering\" event-time associations.\n- The **Event Rectification task** focuses on error diagnosis and correction. The model must first pinpoint temporal logic inconsistencies in a given sequence, then generate a corrected sequence aligned with video content and explain the error's cause. Beyond temporal order comprehension, this task emphasizes mastery of event causality and logical coherence, serving as an evaluation of both \"error-correcting ability\" and \"explanatory capacity\".\n- The **Event Reconstruction task** prioritizes event completion and contextual reasoning. When faced with masked events, the model must infer content consistent with both timeline and scenario plausibility, drawing on logical connections between preceding and subsequent events. This task tests the model's grasp of event flow integrity and its capacity for \"context-driven filling-in reasoning\".\n\nOur intent in designing these three tasks is to **comprehensively evaluate how deeply models understand event temporal logic in long videos through multi-dimensional testing**. A single task might only reflect performance in one scenario, but combining multiple tasks more effectively verifies whether models have truly mastered the dynamics of event associations, strengthening the benchmark's rigor and discriminative power.\n\nWe have incorporated these design rationales into the supplementary material to further clarify the differences among the three tasks.\n\n---\n> **`Q4`:** *\"Although the problems on the EgoMem benchmark involve the understanding of multiple events, the events are relatively independent of each other and lack interactive relationships. Have the authors considered a type of question that requires a comprehensive review of information from multiple events to get an answer, such as: What do the people in event 1 and event 2 accomplish together?\"*\n\n**A:** Thank you for your valuable comments. We fully agree with your view that **the interaction between multiple events is an important dimension in ultra-long video understanding**, as it can more comprehensively evaluate a model's ability to conduct a comprehensive analysis of complex scenarios. In our subsequent work, we plan to address such issues, and some preliminary design ideas are as follows:\n\n- First, we will sort out the timeline and logical connections of different events in the video, and identify event combinations with interactive relationships such as causality, synergy, and contrast. For example, in a video recording of daily work, there is an obvious synergistic relationship between the \"material preparation\" event in the morning and the \"project report\" event in the afternoon.\n- Second, for these event combinations with interactive relationships, we will design diversified types of questions. In addition to the one you mentioned, \"What did the people in Event 1 and Event 2 accomplish together?\", there will also be questions like \"What impacts did the occurrence of Event 1 have on Event 2?\" and \"In what aspects does Event 2 differ from Event 1?\", so as to comprehensively examine the model's ability to understand and integrate information about multi-event interactions.\n\nFollowing your suggestion, we commit to carrying out relevant annotation work in the future and continuously improving the EgoMem benchmark, enabling it to more effectively evaluate the performance of ultra-long video understanding models.\n\n---\n> **`Q5`:** *\"GPT-4o's ability to understand long videos is not its strength. If the proposed method can be compared with a model such as Gemini2.5-pro that has optimized long video understanding ability, it will better illustrate the effectiveness of this method.\"*\n\n**A:** Thank you for your valuable feedback. We fully agree that a comparison with models like Gemini 2.5 Pro, which are specifically optimized for long video understanding, would more comprehensively demonstrate the effectiveness of our proposed method.\n\nWe would like to clarify that due to Gemini 2.5 Pro's public release date of March 25, 2025, its performance data was not yet available on the leaderboards of mainstream long video understanding benchmarks by the NeurIPS 2025 submission deadline of May 15, 2025. Conducting a full evaluation by calling its API on a large number of videos would have incurred a significant time cost. Given these practical constraints, we chose to compare our method with the latest leaderboards and representative SOTA papers updated before the submission deadline, among which GPT4o is a representative leading method.\n\nTo address your concern within the rebuttal period, we conducted an additional, expedited experiment: the Needle-in-A-Video-Haystack test (please see Section 4.2 of our paper for the experimental setup). In this test, we used a 4000-second-long video to evaluate the model's ability to accurately locate and understand key information within extremely long content. The results are as follows:\n- Gemini 2.5 Pro: 70% accuracy\n- VideoLucy (Ours): 80% accuracy\n\nThis result provides preliminary evidence that our VideoLucy framework demonstrates competitive performance, even when compared against a state-of-the-art model optimized for long videos.\n\nFinally, we plan to include the full results of Gemini 2.5 Pro on our newly proposed EgoMem benchmark in a future version of the manuscript. This will provide more comprehensive and convincing experimental support for our conclusions.\n\n---\n> **`Q6`:** *\"The writing of this paper can be further polished (such as the expression of Lines 128-146).\"*\n\n**A:** Thank you for your suggestion. As stated in the first question, we have further polished the expression of this part to make it simpler and understandable.\n\n---\nLast but not least, we would like to sincerely thank Reviewer `Dmx9` again for the valuable time and constructive feedback provided during this review."}, {"review_id": "piddq0bKQs", "reviewer": "Reviewer_4wav", "summary": "The paper proposes VideoLucy, an agent-based framework designed for long video understanding. It introduces a hierarchical memory structure and a deep memory backtracking mechanism that mimic the human recollection process from coarse to fine detail. This system leverages multiple specialized agents (captioning, localization, instruction, and answering agents) to iteratively refine its understanding of a video based on a given question. The authors also introduce EgoMem, a new benchmark built on EgoLife, designed to test both temporal understanding and fine-grained detail perception in long videos. VideoLucy, using open-source models like Qwen2.5-VL and Deepseek-R1, surpasses both open-source and proprietary models on several long video benchmarks such as LVBench, Video-MME, and MLVU, demonstrating strong capabilities in both question answering and temporal reasoning.", "questions": "See Weaknesses.", "limitations": "See Weaknesses.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1.\tAgent-Based Modular Pipeline: The introduction of role-specific agents for captioning, localization, instruction, and answering enables pipelined processing, achieving good performance for video understanding.\n2.\tEgoMem Benchmark: The introduction of EgoMem (Sec. 3) provides an annotated dataset for testing long video understanding at scale.\n\nWeaknesses:\n\n1.\tLimited novelty in agent architecture: While the integration of agents is effective, it builds heavily on existing paradigms without introducing fundamentally new roles or architectural innovations beyond refined prompting.\n2.\tEfficiency concerns: Despite the hierarchical sampling, the iterative backtracking mechanism can be computationally intensive and context-length constrained, especially for ultra-long videos where multiple iterations are required.\n3.\tComparison with prior work: The proposed method performs better than prior agent-based methods across benchmarks. However, there lacks a systematic evaluation on the method component. What are the necessary designs for the proposed method? Does the improvement over prior work primarily come from the usage of latest models (e.g., Qwen2.5-VL).\n4.\tHyperparameters optimized for each benchmark: In L241-242, the temporal scopes were tuned for each single benchmark. How were these parameters are determined? Similarly, how were Ks determined (L139-140)? These components make the proposed method require more engineering tricks.", "was_revised": true, "final_justification": "Thanks the authors for their detailed responses. My concerns are partially addressed and maintain my score. \n\nMore details can be found in the comments below. To summarize, the proposed iterative method requires substantial computation compared to end-to-end VideoLLM, which outweights the (modest) performance gain. Further, the comparison with prior agent-based methods on VideoMME-Long cannot fully manifest where the major improvement come, and incurs an unconvincing gap on EgoMem benchmark. In addition, there are multiple critical hyper-parameters to be tuned manually, which makes the proposed method less appealing.", "rebuttal": "We sincerely thank Reviewer `4wav` for the constructive and thoughtful review. We greatly appreciate your recognition of the proposed agent-based framework, our newly introduced EgoMem benchmark, and the strengths of our empirical results. Below, we respond in detail to your insightful questions and concerns.\n\n---\n> **`Q1`:** *\"While the integration of agents is effective, the framework builds on existing paradigms without introducing fundamentally new roles beyond refined prompting.\"*\n\n**A:** Thank you for your valuable feedback regarding the agent architecture. We appreciate the chance to clarify **how our work advances beyond refined prompting** -- a distinction central to our contributions.\n\nWhile agent-based systems for long video understanding do share high-level iterative reasoning paradigms, **VideoLucy introduces fundamental architectural innovations in memory mechanism design** that distinguish it from existing systems. As highlighted in our abstract, the core innovation lies in the **hierarchical memory structure with progressive granularity**, which explicitly models detail levels and temporal scopes across hierarchical depths. This structure is **not merely a prompt engineering technique but a structural redesign** of how agents store, retrieve, and integrate video information over time.\n\nSpecifically, our **deep memory backtracking mechanism** represents a departure from conventional agent workflows. Unlike existing systems that either operate on flat memory buffers or fixed temporal windows, VideoLucy's agent employs a systematic backtracking process inspired by human recollection -- starting from coarse temporal contexts and iteratively drilling down into finer-grained details. This requires the agent to dynamically adjust memory access patterns based on accumulated information, a capability that not only stems from the hierarchical memory's explicit encoding of temporal relationships (rather than prompt heuristics alone) but also endows the agent with **a novel \"memory coordinator\" role**: unlike existing agents that function solely as \"information processors\", ours actively plans the timing and granularity of memory retrieval, a role unmodeled in prior paradigms.\n\n**Our experimental results (Figure `6`) further validate this architectural novelty:** VideoLucy's performance gains stem from the synergy between its memory hierarchy and iterative backtracking, not just prompt refinements. For instance, on the Video-MME long split, we observe that ablating the hierarchical memory structure (while keeping prompting strategies identical) leads to a 24.7% drop in accuracy, confirming the structural contribution of our memory mechanism.\n\nWe acknowledge that prompting coordinates agent behaviors, but it **serves as an interface** to execute the underlying architectural innovations **rather than being the primary source** of novelty. Our innovative design of VideoLucy **provides a new and effective approach** to long video understanding, making a valuable contribution to the community.\n\nWe will strengthen this distinction to better highlight these structural advancements.\n\n---\n> **`Q2`:** *\"Efficiency concerns: Despite the hierarchical sampling, the iterative backtracking mechanism can be computationally intensive and context-length constrained, especially for ultra-long videos where multiple iterations are required.\"*\n\n**A:** Thank you for your valuable feedback regarding the computational intensity and context-length constraints of the iterative backtracking mechanism, especially for ultra-long videos.\n\nAs for these two aspects, we would like to draw your attention to the Section `A.2` Limitation (on Page `15`) of our manuscript, where they are already discussed. We elaborate again as follows for your convenient reference:\n- Regarding the **computational overhead**, we acknowledge that agent-based long video understanding systems, including ours, generally require multiple iterations to achieve accurate answers, which indeed results in higher computational costs compared to traditional end-to-end video MLLMs. However, considering that the memories obtained in our framework can be stored, for multiple different questions about the same video, we can fully leverage the video's pre-existing memories to achieve faster responses, instead of repeatedly performing inference on the video's information, as existing video MLLMs do. This can reduce the time overhead of reasoning to a certain extent, which is conducive to deployment in practical applications.\n\n- As for the **context-length constraint**, the maximum video length that VideoLucy can process is currently bounded by the LLM's maximum context limit, which poses a challenge for processing super-long videos exceeding existing benchmarks (e.g., hundreds of hours). Nevertheless, since VideoLucy is a training-free framework, it can seamlessly integrate with any LLM with enhanced capabilities. As LLM technology advances, we anticipate steady improvements in both the performance of VideoLucy and the maximum video length it can handle. Notably, despite this constraint, VideoLucy has achieved leading experimental results on our EgoMem benchmark, where the average video duration is **6.33 hours** -- one of the longest among existing benchmarks to date. This demonstrates that our framework is already capable of effectively handling extremely long videos within the current context-length limits of state-of-the-art LLMs.\n\n---\n> **`Q3`:** *\"The proposed method performs better than prior agent-based methods across benchmarks. However, there lack of an evaluation on the component. What are the necessary designs for the proposed method? Does the improvement over prior work primarily come from the usage of the latest models?\"*\n\n**A:** Thank you for your attention to the systematic evaluation of our method components and the source of improvements. We would like to address this from two aspects:\n- On one hand, we **have conducted ablation studies on the core components** of VideoLucy, namely the hierarchical memory structure and iterative backtracking mechanism, as presented in our paper. In Figure `5`, we verify that both the richness of memory information and its relevance to the question gradually increase during the memory backtracking process. Additionally, in Figure `6`, we investigate the \"Effects of different memory depth and maximum iteration number\", which validates the effectiveness of the hierarchical memory structure and helps us select a trade-off number of iterations. Admittedly, similar to previous methods, since VideoLucy is an integrated agent system and long video understanding is a resource-intensive task, we have not performed ablation experiments on each role-specific agent within the system. However, the experiments in Figures `5` and `6` sufficiently demonstrate the effectiveness of the core designs of VideoLucy.\n- On the other hand, to ensure fair comparisons, when we reproduced the prior agent-based methods (VideoAgent and VideoTree) on our EgoMem benchmark, we **kept the MLLM and LLM consistent** with those used in VideoLucy, specifically Qwen2.5-VL-7B and DeepSeek-R1. As shown in Table `4`, VideoLucy outperforms these two methods by 25.2% and 23.6% in accuracy, respectively. This fair setting clearly indicates that the improvement stems from our method's design rather than the use of the latest models. We will emphasize such a consistent and fair setup in the experimental explanations.\n\nWe believe these experiments adequately validate the necessity and effectiveness of our key designs, including the hierarchical memory structure and iterative backtracking mechanism.\n\n---\n> **`Q4`:** *\"Hyperparameters optimized for each benchmark: In L241-242, the temporal scopes were tuned for each single benchmark. How were these parameters determined? Similarly, how were Ks determined (L139-140)?\"*\n\n**A:** Thank you for your concern regarding the determination of hyperparameters for each benchmark. \n\nThe tuning of temporal scopes for different benchmarks is primarily driven by the **significant variations in video durations**. Using a fixed temporal scope across all benchmarks would be impractical, as illustrated by the following example: a coarse-grained memory scope of 50 seconds would result in only 2 coarse memories for a 100-second video but 72 coarse memories for a 3600-second video. \n\nWe conducted preliminary qualitative experiments on several videos of varying lengths and found that too few coarse memories would limit the initial information available during iterative backtracking, thereby increasing the number of iterations required for memory search. Conversely, an excessive number of coarse memories would introduce redundant information, unnecessarily expand the context length, and interfere with effective memory retrieval.\n\nTo address this, we determined the temporal scopes based on a principle that ensures the number of coarse-grained memories **falls within a reasonable range** (typically 10–36) across all benchmarks. Additionally, we maintained a consistent hierarchical relationship where the scope of finer-grained memories is set to approximately one-tenth of the scope of the level above. This design balances the trade-off between initial information richness and context length efficiency, while preserving the hierarchical memory structure's integrity.\n\nDetailed values of the temporal scopes and Ks for each benchmark are provided on Page `1` of the supplementary material. We acknowledge that this benchmark-specific tuning involves empirical considerations, but we emphasize that it is guided by **clear principles** derived from the structural properties of the datasets (e.g., video duration, event density). We have added a detailed explanation of this part in the supplementary material.\n\n---\nLast but not least, we would like to sincerely thank Reviewer `4wav` again for the valuable time and constructive feedback provided during this review."}, {"review_id": "hxmlEAWC1l", "reviewer": "Reviewer_YuVT", "summary": "Authors introduce a memory backtracking framework for long video understanding inspired by human recollection process. They combine a hierarchical memory structure with iterative agent-based search to capture both temporal relations as well as fine-grained details. The authors also propose a new benchmark, EgoMem for better evaluating this aspect of models. Extensive evaluations on existing benchmarks as well as EgoMem are performed to establish usefulness of their method.", "questions": "Consider discussing prior memory based methods for long video understanding such as [1,2]\n\n[1] Kahatapitiya et al. “Language Repository for Long Video Understanding.” \n\n[2] Ryoo at al, \"Token Turing Machines\"", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. Clear writing and presentation with good use of figures \n2. Multiple qualitative evaluations present insights into model behavior \n3. Strong results with consistent improvements across benchmarks\n4. Novel and interesting exploration into hierarchical memory structure\n5. The needle in the hay-stack experiment is particularly interesting\n\n**Weaknesses**\n1. Is backtracking sensitive to the initial memory list (Alg 1 L1) created?\n2. What is the inference time for this method? How does this compare to other agent based works?\n\n\nUpdated Version (upon request for longer review)\n-------------------\nStrengths\n1. Clear writing and presentation with good use of figures\n\nDiscussion of prior work drawbacks in terms of limited cross-frame temporal context modeling and sparse frame sampling is used to clearly draw motivation.\nAlgorithm 1 gives a full overview of the method describing it concisely and clearly.\nAblative studies in experiment section are well described and analyzed. For example, two ablations that specifically explore aspects of their problem statement, \"Needle-in-A-Video-Haystack\" and \"Information Richness\", are explained in detail (in terms of implementation) and the obtained conclusions from results are clearly highlighted.\nFigure 2 gives a clear overview of the hierarchical memory idea. In terms of the proposed dataset, Figure 3 gives a decent overview illustrating the 6 aspects evaluated in the benchmark.\n2. Multiple qualitative evaluations present insights into model behavior\n\nFigure 7 & 8 give qualitative insights to how proposed method operates on specific examples.\nThe comparison to baselines highlights the kind of tasks proposed method is supposed to solve better and what drawbacks authors try to overcome.\n3. Strong results with consistent improvements across benchmarks\n\nLVBench, MLVU, and Video-MME are established, large-scale benchmarks for long video QnA. The long split of Video-MME (41 mins average length with some videos over 1 hour long) is specifically challenging. The proposed method outperforms even closed source models like Gemini and GPT4o on these benchmarks. Additionally, they report SOTA results across all three of these challenging benchmarks.\nIn LVBench, improvements are consistent across all sub-categories.\nOn Video-MME, large improvements for long split (avg len 41 mins), moderate improvements for medium split, and on-par results for short split, highlighting the arguments of how memory backtracking solves issues faced by prior work in longer video cases (i.e. larger improvement for longer videos).\n4. Novel and interesting exploration into hierarchical memory structure.\n\nHierarchical key-frame selection (e.g. [32, 45, 46] in paper) have been demonstrated to be useful for long-video QnA, but not hierarchical memory (to the best of this reviewers knowledge). This makes the study (on using agents to further retrieve information from important parts as opposed to simply selecting good frames to caption) both novel and interesting.\nMemory structures have also been explored in long video QnA settings (e.g. [1, 2] in original review). But not hierarchical to the best of this reviewers knowledge. This also makes the proposed direction novel and interesting.\n5. The needle in the hay-stack experiment is particularly interesting.\n\nL298 proposes an ablation on artificially construction videos (insert short clip into a longer video) where models are queried about those inserted short clips. This provides a targeted evaluation for the authors preliminary motivation (drawback of sparse frame sampling based prior work).\nThe ablation evaluates total videos lengths over a range from few minutes to over one hour, validating the effectiveness of proposed method in solving the identified drawback across videos of varying lengths.\n \n\nWeaknesses\n1. Is backtracking sensitive to the initial memory list (Alg 1 L1) created?\n\nRight after L184 in paper, there is algorithm 1. On the first line of this, sparse coarse memory is initialized. How sensitive is the overall algorithm to this init operation? Does the loop on Alg L4 run longer for bad init? Or shorter for good init?\n2. What is the inference time for this method? How does this compare to other agent based works?\n\nPlease provide clear inference time calculations for the full method as well as its sub-components.\nSince the method is iterative, consider calculating mean and standard deviations. Also discuss what kind of video-question pairs cause higher inference time vs lower? For example, which tasks are harder for this algorithm, and require more inference compute?\nIt will be useful to compare the end-to-end inference time for some dataset against 2-3 prior works that do similar hierarchical / iterative operations (e.g. VideoTree).", "was_revised": true, "final_justification": "Only minor clarifications were requested, and the authors answer these clearly. \n\nOther reviewers share similar rating. An issue on inference speed is raised, but authors address this well. \n\nRetaining vote for accept.", "rebuttal": "We sincerely thank Reviewer `YuVT` for the positive and encouraging review. We are grateful for your recognition of our hierarchical memory design, detailed experimental results, and the quality of writing and visualizations. Below, we respond to your thoughtful questions and suggestions in detail.\n\n---\n> **`Q1`:** *\"Is backtracking sensitive to the initial memory list (Alg. 1 L1) created?\"*\n\n**A:** Thank you for your insightful question regarding whether backtracking is sensitive to the initial memory list.\n\nTheoretically, **the backtracking process is sensitive to the initial memory list**, as the iterative deepening of memory is built upon the time periods and content in the initial list. However, in practical scenarios, **this sensitivity rarely affects the final performance**, which can be attributed to the following aspects:\n\n- First, our initial memory list is inherently question-aware. As described in Lines 185-194, the initial list $CM_\\mathrm{sinit}$ is derived by filtering all the sampled coarse memories through the localization agent, which specifically identifies time periods most relevant to the question. This ensures that the initial memory list is closely associated with the query from the outset, laying a reliable foundation for subsequent backtracking.\n- Second, the strong text understanding and processing capabilities of LLMs play a crucial role. In our qualitative experiments, we observed that the localization agent, powered by LLMs (e.g., DeepSeek-R1), rarely fails to identify relevant time periods entirely. Even in cases where the initial memory list may have minor inaccuracies, the iterative backtracking mechanism can gradually correct and refine the memory by mining deeper details, thereby mitigating potential impacts.\n\nIn summary, while the backtracking process is theoretically sensitive to the initial memory list, the question-relevant initialization strategy and the robustness of LLMs ensure that, in practice, the initial memory list is constructed accurately enough to avoid significant adverse effects on the backtracking process.\n\n---\n> **`Q2`:** *\"What is the inference time for this method? How does this compare to other agent-based works?\"*\n\n**A:** Thank you for raising this important point. We appreciate the opportunity to clarify the inference time of VideoLucy and how it compares to other agent-based approaches.\n\nBy default, our VideoLucy employs Qwen2.5-VL-7B as the MLLM and DeepSeek-R1 as the LLM. The inference time overhead mainly stems from two aspects: the MLLM performing inference on short video clips and the LLM conducting inference on long texts. Practically, both can be accelerated using various inference acceleration frameworks to enhance the inference speed.\n\nIn our experiments, we deployed Qwen2.5-VL-7B locally on 8 A100 GPUs using VLLM. For **a one-hour video**, through batch inference, we can obtain text descriptions of each clip within **a dozen seconds**. As for the LLM, due to the high cost of local deployment, we utilized the official API. Although the API call time fluctuates, making it challenging to provide accurate inference time under fair and reproducible conditions, our practical observations show that for a one-hour video, VideoLucy can generally generate accurate answers with **about five API calls**.\n\nFor fair comparison with other agent-based systems, a feasible approach is to **compare the number of LLM calls**. This metric avoids the interference of hardware configurations and deployment environments, providing a more objective reference for comparing the efficiency of different agent-based methods.\n\nWe have conducted a quick experiment to briefly compare the number of LLM calls. The experimental setup was as follows: one question-answer pair was randomly selected from each of the 42 videos in EgoMem to form a test set. Then, three agent systems -- VideoAgent, VideoTree, and VideoLucy -- were tested, with the maximum number of LLM calls limited to 20. \n\nThe test results showed that the average number of calls required for the three systems was **14.1, 9.6, and 6.3** respectively. This indicates that our VideoLucy has an advantage over other methods in terms of inference speed.\n\n---\n> **`Q3`:** *\"Consider discussing prior memory-based methods for long video understanding.\"*\n\n**A:** Thank you for your valuable recommendation of these two seminal works in long video understanding.\n\n*LangRepo* [`R1`] pioneers a framework for long video understanding by maintaining a structured language repository tailored for LLMs. By iteratively updating multi-scale video chunks and pruning redundant text through write/read operations, it efficiently captures temporal information while preserving interpretability -- effectively translating video content into a language-aligned format that better fits LLM processing paradigms. Its state-of-the-art zero-shot performance on benchmarks like EgoSchema and NExT-QA highlights its success in balancing long-term context modeling and computational efficiency.\n\n*Token Turing Machines (TTM)* [`R2`] presents a seminal contribution to long video understanding by introducing an efficient memory-augmented Transformer framework. Inspired by Neural Turing Machines, TTM integrates an external memory module that dynamically summarizes historical frames into tokens, enabling bounded computational complexity while processing long sequences. This design elegantly addresses the scalability challenge inherent in standard Transformers, as each new observation interacts only with memory tokens rather than the entire history. \n\nOur VideoLucy, as a memory-based framework, builds on these advancements: we extend their core ideas via hierarchical memory structures and agent-driven backtracking, thereby enhancing recall granularity for complex long videos requiring fine-grained details.\n\nWe have incorporated a detailed discussion of these two works in the revised manuscript.\n\n---\n\n**References:**\n- [`R1`] Kahatapitiya, K., Ranasinghe, K., Park, J., & Ryoo, M. S. (2024). \"Language Repository for Long Video Understanding.\" arXiv preprint arXiv:2403.14622.\n- [`R2`] Ryoo, M. S., Gopalakrishnan, K., Kahatapitiya, K., Xiao, T., Rao, K., Stone, A., ... & Arnab, A. (2023). \"Token Turing Machines.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 19070-19081).\n\n---\nLast but not least, we would like to sincerely thank Reviewer `YuVT` again for the valuable time and constructive feedback provided during this review."}]

Author rebuttal:

Meta-review: ### Summary
The paper introduces VideoLucy, a novel framework for long video understanding. VideoLucy operates by creating memories of the video at progressive granularities (from coarse to fine) and uses a set of coordinated agents to iteratively "backtrack" and mine deeper, question-relevant details until a confident answer can be formulated. Extensive experiments have been conducted on multiple long video understanding benchmarks (VideoMME, MLVU, LVBench), showing that their method significantly outperforms SOTA. Additionally, they introduce a new benchmark, EgoMem, specifically designed to evaluate a model's ability to reason about complex temporal events and fine-grained details in extremely long videos.

### Strengths
- Novel framework and motivation: The proposed memory backtracking mechanism is novel and well-motivated. The hierarchical memory structure is a creative and interesting approach to managing information at different temporal scales (noted by reviewers RcwM, Dmx9, and YuVT).
- Strong empirical performance: The paper presents compelling results, demonstrating state-of-the-art performance across several challenging long video benchmarks. (noted by reviewers RcwM, Dmx9, and YuVT).
- A new long-form video understanding benchmark: The introduction of the EgoMem benchmark is a significant contribution to the community. It addresses a gap in existing evaluation by focusing on fine-grained temporal reasoning in ultra-long videos (noted by Reviewer 4wav).
- Clarity and presentation: The paper is generally well-written, with high-quality illustrations and clear explanations of its complex architecture, making the contributions easy to understand (noted by Reviewers RcwM and YuVT).

### Weaknesses of the Paper
- Efficiency and computational cost: A major concern, raised consistently across reviews (Reviewers 4wav, YuVT), is the computational cost and inference latency of the iterative, agent-based approach. 
- Hyperparameter sensitivity: The framework requires several key hyperparameters (e.g., temporal scopes for memory levels) to be manually tuned for each specific benchmark. ( Reviewer 4wav).
- Novelty of agent architecture: While the memory structure is novel, some of the underlying agent architecture builds on existing paradigms. (Reviewer 4wav). But this is not supported by any reference paper.
- Ablation and component analysis: While ablation studies were provided, Reviewer 4wav noted that it was difficult to fully disentangle the performance gains from the powerful base models (e.g., Qwen2.5-VL) versus the architectural contributions of VideoLucy itself.

### Reasons for Recommendation
I am recommending this paper for acceptance as a poster.

The decision is based on the consensus that the paper's strengths, particularly its novel memory backtracking concept and strong empirical results, outweigh its weaknesses. I agree with most of the reviewers that this paper proposes a new way of structuring and retrieving information for long video reasoning, which is a significant contribution. The impressive performance on multiple difficult benchmarks validates the effectiveness of this approach. Furthermore, the release of the EgoMem benchmark provides a valuable and timely resource for the research community.

However, the paper is not recommended for a spotlight or oral presentation due to the valid concerns raised about computational efficiency and the practical challenges of hyperparameter tuning.

### Summary of Discussion and Rebuttal
The authors provided a strong rebuttal and successfully addressed most of the concerns. 

- The authors answered all the clarification questions about the mathematical notation, and provided a principled guide for setting hyperparameters based on video duration.
- More experimental results are provided. The authors added a runtime analysis table, included direct comparisons with the base MLLMs to show the framework's added value, and conducted a new "Needle-in-a-Haystack" experiment against Gemini 2.5 Pro, showing competitive performance.
- In response to Reviewer YuVT, they compared the average number of LLM calls to other agent systems to demonstrate better efficiency in that regard and added a discussion of related memory-based works.
- For the EgoMem benchmark, they provided a strong rationale for the task design and acknowledged plans to include multi-event interaction questions in future work.

After the discussion, Reviewers RcwM and YuVT are satisfied by the rebuttal and maintained their "Accept" recommendations. Reviewer Dmx9's concerns were also resolved, leading them to maintain a "Borderline accept" score. Reviewer 4wav are still not fully convinced on the points of efficiency. Overall, the authors' thorough rebuttal strengthened the paper and convinced the majority of reviewers towards acceptance. The remaining concerns, while valid, do not undermine the paper's core contributions.
Decision: Accept (poster)