Title: VideoLucy: Deep Memory Backtracking for Long Video Understanding
Venue: NeurIPS 2025
Accepted: yes
Keywords: Long Video Understanding

Agent-based System
Abstract: Recent studies have shown that agent-based systems leveraging large language models (LLMs) for key information retrieval and integration have emerged as a promising approach for long video understanding. However, these systems face two major challenges. First, they typically perform modeling and reasoning on individual frames, struggling to capture the temporal context of consecutive frames. Second, to reduce the cost of dense frame-level captioning, they adopt sparse frame sampling, which risks discarding crucial information. To overcome these limitations, we propose VideoLucy, a deep memory backtracking framework for long video understanding. Inspired by the human recollection process from coarse to fine, VideoLucy employs a hierarchical memory structure with progressive granularity. This structure explicitly defines the detail level and temporal scope of memory at different hierarchical depths. Through an agent-based iterative backtracking mechanism, VideoLucy systematically mines video-wide, question-relevant deep memories until sufficient information is gathered to provide a confident answer. This design enables effective temporal understanding of consecutive frames while preserving critical details. In addition, we introduce EgoMem, a new benchmark for long video understanding. EgoMem is designed to comprehensively evaluate a model's ability to understand complex events that unfold over time and capture fine-grained details in extremely long videos. Extensive experiments demonstrate the superiority of VideoLucy. Built on open-source models, VideoLucy significantly outperforms state-of-the-art methods on multiple long video understanding benchmarks, achieving performance even surpassing the latest proprietary models such as GPT-4o. Our code and dataset will be made publicly available.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAuthors introduce a memory backtracking framework for long video understanding inspired by human recollection process. They combine a hierarchical memory structure with iterative agent-based search to capture both temporal relations as well as fine-grained details. The authors also propose a new benchmark, EgoMem for better evaluating this aspect of models. Extensive evaluations on existing benchmarks as well as EgoMem are performed to establish usefulness of their method.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. Clear writing and presentation with good use of figures \n2. Multiple qualitative evaluations present insights into model behavior \n3. Strong results with consistent improvements across benchmarks\n4. Novel and interesting exploration into hierarchical memory structure\n5. The needle in the hay-stack experiment is particularly interesting\n\n**Weaknesses**\n1. Is backtracking sensitive to the initial memory list (Alg 1 L1) created?\n2. What is the inference time for this method? How does this compare to other agent based works?\n\n\nUpdated Version (upon request for longer review)\n-------------------\nStrengths\n1. Clear writing and presentation with good use of figures\n\nDiscussion of prior work drawbacks in terms of limited cross-frame temporal context modeling and sparse frame sampling is used to clearly draw motivation.\nAlgorithm 1 gives a full overview of the method describing it concisely and clearly.\nAblative studies in experiment section are well described and analyzed. For example, two ablations that specifically explore aspects of their problem statement, \"Needle-in-A-Video-Haystack\" and \"Information Richness\", are explained in detail (in terms of implementation) and the obtained conclusions from results are clearly highlighted.\nFigure 2 gives a clear overview of the hierarchical memory idea. In terms of the proposed dataset, Figure 3 gives a decent overview illustrating the 6 aspects evaluated in the benchmark.\n2. Multiple qualitative evaluations present insights into model behavior\n\nFigure 7 & 8 give qualitative insights to how proposed method operates on specific examples.\nThe comparison to baselines highlights the kind of tasks proposed method is supposed to solve better and what drawbacks authors try to overcome.\n3. Strong results with consistent improvements across benchmarks\n\nLVBench, MLVU, and Video-MME are established, large-scale benchmarks for long video QnA. The long split of Video-MME (41 mins average length with some videos over 1 hour long) is specifically challenging. The proposed method outperforms even closed source models like Gemini and GPT4o on these benchmarks. Additionally, they report SOTA results across all three of these challenging benchmarks.\nIn LVBench, improvements are consistent across all sub-categories.\nOn Video-MME, large improvements for long split (avg len 41 mins), moderate improvements for medium split, and on-par results for short split, highlighting the arguments of how memory backtracking solves issues faced by prior work in longer video cases (i.e. larger improvement for longer videos).\n4. Novel and interesting exploration into hierarchical memory structure.\n\nHierarchical key-frame selection (e.g. [32, 45, 46] in paper) have been demonstrated to be useful for long-video QnA, but not hierarchical memory (to the best of this reviewers knowledge). This makes the study (on using agents to further retrieve information from important parts as opposed to simply selecting good frames to caption) both novel and interesting.\nMemory structures have also been explored in long video QnA settings (e.g. [1, 2] in original review). But not hierarchical to the best of this reviewers knowledge. This also makes the proposed direction novel and interesting.\n5. The needle in the hay-stack experiment is particularly interesting.\n\nL298 proposes an ablation on artificially construction videos (insert short clip into a longer video) where models are queried about those inserted short clips. This provides a targeted evaluation for the authors preliminary motivation (drawback of sparse frame sampling based prior work).\nThe ablation evaluates total videos lengths over a range from few minutes to over one hour, validating the effectiveness of proposed method in solving the identified drawback across videos of varying lengths.\n \n\nWeaknesses\n1. Is backtracking sensitive to the initial memory list (Alg 1 L1) created?\n\nRight after L184 in paper, there is algorithm 1. On the first line of this, sparse coarse memory is initialized. How sensitive is the overall algorithm to this init operation? Does the loop on Alg L4 run longer for bad init? Or shorter for good init?\n2. What is the inference time for this method? How does this compare to other agent based works?\n\nPlease provide clear inference time calculations for the full method as well as its sub-components.\nSince the method is iterative, consider calculating mean and standard deviations. Also discuss what kind of video-question pairs cause higher inference time vs lower? For example, which tasks are harder for this algorithm, and require more inference compute?\nIt will be useful to compare the end-to-end inference time for some dataset against 2-3 prior works that do similar hierarchical / iterative operations (e.g. VideoTree).\n\nQUESTIONS:\nConsider discussing prior memory based methods for long video understanding such as [1,2]\n\n[1] Kahatapitiya et al. “Language Repository for Long Video Understanding.” \n\n[2] Ryoo at al, \"Token Turing Machines\"\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes VideoLucy, an agent-based framework designed for long video understanding. It introduces a hierarchical memory structure and a deep memory backtracking mechanism that mimic the human recollection process from coarse to fine detail. This system leverages multiple specialized agents (captioning, localization, instruction, and answering agents) to iteratively refine its understanding of a video based on a given question. The authors also introduce EgoMem, a new benchmark built on EgoLife, designed to test both temporal understanding and fine-grained detail perception in long videos. VideoLucy, using open-source models like Qwen2.5-VL and Deepseek-R1, surpasses both open-source and proprietary models on several long video benchmarks such as LVBench, Video-MME, and MLVU, demonstrating strong capabilities in both question answering and temporal reasoning.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.\tAgent-Based Modular Pipeline: The introduction of role-specific agents for captioning, localization, instruction, and answering enables pipelined processing, achieving good performance for video understanding.\n2.\tEgoMem Benchmark: The introduction of EgoMem (Sec. 3) provides an annotated dataset for testing long video understanding at scale.\n\nWeaknesses:\n\n1.\tLimited novelty in agent architecture: While the integration of agents is effective, it builds heavily on existing paradigms without introducing fundamentally new roles or architectural innovations beyond refined prompting.\n2.\tEfficiency concerns: Despite the hierarchical sampling, the iterative backtracking mechanism can be computationally intensive and context-length constrained, especially for ultra-long videos where multiple iterations are required.\n3.\tComparison with prior work: The proposed method performs better than prior agent-based methods across benchmarks. However, there lacks a systematic evaluation on the method component. What are the necessary designs for the proposed method? Does the improvement over prior work primarily come from the usage of latest models (e.g., Qwen2.5-VL).\n4.\tHyperparameters optimized for each benchmark: In L241-242, the temporal scopes were tuned for each single benchmark. How were these parameters are determined? Similarly, how were Ks determined (L139-140)? These components make the proposed method require more engineering tricks.\n\nQUESTIONS:\nSee Weaknesses.\n\nLIMITATIONS:\nSee Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an agent-based long video understanding method VideoLucy, which imitates the human memory recall mechanism from coarse to fine. In order to verify the effectiveness of the proposed method, this paper also conducts comprehensive evaluation on three commonly used long video understanding datasets such as MLVU, VideoMME and LVbench and the newly collected EgoMem Benchmark. The results on all benchmarks show the effectiveness of VideoLucy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. This paper imitates the human memory mechanism to design a video understanding agent from coarse to fine, and the research motivation is reasonable.\n2. The performance advantage of the proposed method is significant, and the long video understanding ability exceeds the advanced models in the industry such as GPT-4o.\n3. The long video understanding ability verification experiments are comprehensive.\n\nWeaknesses:\n1. Is there a problem with the $y_i = k$ condition in the Line 132? In this way, only the first K video frames can be sampled, and each frame is a clip?\n2. The method in this paper needs to design different hyperparameters for different datasets. Can some general design references be given?\n3. Among the 6 types of event understanding tasks in the EgoMem benchmark, Event Order, Event Rectification, and Event Reconstruction essentially test the same ability of the model. What is the basis for this split?\n4. Although the problems on the EgoMem benchmark involve the understanding of multiple events, the events are relatively independent of each other and lack interactive relationships. Have the authors considered a type of question that requires comprehensive review of information from multiple events to get an answer, such as: What do the people in event 1 and event 2 accomplish together?\n5. GPT-4o's ability to understand long videos is not its strength. If the proposed method can be compared with a model such as Gemini2.5-pro that has optimized long video understanding ability, it will better illustrate the effectiveness of this method.\n\nQUESTIONS:\nIn general, I think the research motivation and method design of this paper are reasonable, but there are some aspects that can be improved, such as: 1) the description of the method details can be clearer (weaknesses 1 and 2); 2) the type of questions set for the benchmark can be more diverse (weakness 3 and 4); 3) the performance comparison can be more comprehensive (weakness 5). In addition, the writing of this paper can be further polished (such as the expression of Lines 128-146).\n\nLIMITATIONS:\nNo negative social impact has been found so far.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces VideoLucy, a deep memory backtracking framework for long video understanding that leverages a hierarchical memory structure and agent-based iterative backtracking to effectively capture temporal context and preserve fine details. By progressively mining question-relevant information, VideoLucy overcomes the limitations of sparse frame sampling and improves reasoning over long video sequences. Extensive experiments and a newly proposed EgoMem benchmark demonstrate that VideoLucy outperforms state-of-the-art and even proprietary models like GPT-4o on several long video understanding tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\nThe idea presented in this paper is novel and holds significant potential for practical applications in video tasks. The paper is clearly written and is supported by comprehensive experiments, which leaves a strong impression. The structure is well-organized, and the illustrations are of high quality.\n\n**Weaknesses**\n\nWhile the paper presents promising results, it lacks sufficient clarity in its introduction and explanation of key concepts, which may hinder understanding for readers unfamiliar with certain references. Additionally, the methodology’s reliance on uniform video segmentation may introduce limitations, and important details regarding experimental baselines, model choices, and resource requirements are insufficiently discussed. A more comprehensive analysis of these aspects would strengthen the work.\n\nQUESTIONS:\n**A. About the Writing**\n\n**A.1** Please consider revising the introduction's first paragraph, especially the section on video understanding. For readers unfamiliar with the movie \"Lucy,\" it is difficult to grasp the context of the paper.\n\n**A.2** The concept of the \"human recollection process from coarse to fine\" is mentioned but not explained in either the abstract or the introduction. Providing a clear explanation or an example would help the reader better understand this idea.\n\n**B. About the Methodology**\n\n**B.1** The method divides videos into equally sized clips, which may cause important events to be split across multiple clips. Have you considered alternative segmentation methods or analyzed the potential drawbacks of uniform segmentation?\n\n**C. About the Experiment**\n\n**C.1** What are the runtime and memory requirements of your method? Including a chart or table to illustrate these aspects would be helpful.\n\n**C.2** Why do the baselines differ in Tables 1, 2, and 3? Is this due to certain baselines being incompatible with some datasets, or were they specifically selected by the authors?\n\n**C.3** It is unclear which models are used in CapAGT, LocAGT, and other variants. It would be better to specify the base models used and compare your approach with their baseline capabilities, ideally through experiments using different underlying models.\n\n**C.4** The evaluation on EgoMem shows that your method has a significant advantage in temporal understanding. Can you clarify which aspects of your approach contribute most to this improvement?\n\n**C.5** Please analyze the improvements achieved on different tasks. Since the proposed benchmarks in the paper are primarily focused on temporal understanding, is the performance gain on benchmarks such as Video-MME and LVBench mainly attributable to enhanced temporal sequence understanding?\n\nLIMITATIONS:\nThe authors adequately addressed the limitations and potential negative societal impact of their work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}