Memento: Toward an All-Day Proactive Assistant for Ultra-Long Streaming Video
Authors: Hongxiang Jiang
Zengrui Ge
guo chen
Qixiong Wang
Jile Jiao
Xuetao Feng
Yuan Wang
Yan Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Vision-Language Models; Online Ultra-Long video understanding; Dynamic Memory
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Multimodal large language models have demonstrated impressive capabilities in visual-language understanding, particularly in offline video tasks. More recently, the emergence of online video modeling has introduced early forms of active interaction. However, existing models, typically limited to tens of minutes, are not yet capable of all-day proactive understanding over ultra-long video streams. They struggle to maintain long-term context online, as they suffer from token accumulation and lack scalable memory mechanisms. These limitations hinder critical tasks such as reminding users that medication was taken hours earlier—an ability that exemplifies the shift from reactive to memory-oriented assistants with long-term reasoning. To bridge this gap, we present Memento, the first proactive vision-language framework for ultra-long streaming video. To avoid token growth and support scalable long-duration understanding, we introduce Dynamic Memory and Query-related Memory Selection, enabling sparse memory retention and efficient retrieval. To address the training challenges of memory-based modeling, we propose Step-Aware Memory Attention, which aligns memory access with temporal steps for stable supervision. To support both training and evaluation of active, long-term behavior, we construct Memento-54K and MementoBench, a dataset-benchmark suite covering diverse tasks on text, object, and action across video streams up to 7 hours. Experiments demonstrate that Memento achieves superior performance, paving the way toward reliable all-day proactive video assistants.

Published as a conference paper at ICLR 2026
MEMENTO: TOWARD AN ALL-DAY PROACTIVE ASSIS- TANT FOR ULTRA-LONG STREAMING VIDEO
Hongxiang Jiang'
Zengrui Ge
Guo Chen
Qixiong Wang*
Jile Jiao'
Xuetao Feng' Yuan Wang
Yan Wang*1
1 Deepeleph Intelligent Technology
Henan Univeristy
BBeijing University of Aeronautics and Astronautics wy843780 ieniiiimeme come..cm.commm
*Xiaohongshu
TTinghua University
ABSTRACT
Multimodal large language models have demonstrated impressive capabilities in visual-language understanding, particularly offline video tasks. More recently, the emergence of online video modeling has introduced early forms of active interaction. However, existing models, typically limited to tens of minutes, are not yet capable of all-day proactive understanding over ultra-long video streams. They struggle to maintain long-term context online, as they suffer from token accumulation and lack scalable memory mechanisms. These limitations hinder critical tasks such as reminding users that medication was taken hours carlier-an ability that exemplifies the shift from reactive to memory-oriented assistants with long-term reasoning. To bridge this gap, we present Memento, the first proactive vision Laiiunla framework for ultra-long streaming video. To avoid token growth and support scalable long- duration understanding, we introduce Dynamic Memory and Query-related Memory Selection, enabling sparse memory retention and effi cient retrieval. To address the training challenges of memory-based modeling, we propose Step-Aware Memory Attention, which aligns memory access with tempo- ral steps for stable supervision. To support both training and evaluation of active, long-term behavior, we construct Memento-544 55K and MementoBench, dataset- benchmark suite covering diverse tasks text, object, and action across video streams up to 7 hours. Experiments demonstrate that Memento achieves superior performance, paving the way toward reliable all-day proactive video assistants.
INTRODUCTION
Recent advancements large language models (LLMs)
(uyang et al. 2022 Touvron al. 2023
Yang et al. 2024b a Xin et al. 2025 Guo et al. 2025 and vision-language models (VLMs) (Liu
ct al. 2023 2024a; Achiam et al. 2023 Bai et al. 22025) have shown remarkable progress in video
understanding, particularly with the emergence of long form (Ren et al.) 2024 Song et al. 2024a
Zeng et al. 2025) and online video LLMs (Chen et al. 2024a: Wu et al. 22024b Li et al. 2025a
Qian et al. 2025). Such progress has further raised expectations for all- proactive assistant.
This assistant would continuously perceive the environment through ultra-long video streams and proactively interact with humans, rather than merely responding passively explicit user queries. Achieving this capability would not only fundamentally transform the role of Al assistants in daily human activities, but also represent a critical step toward genuine autonomous agents (Fan al.
2024
Wang et al. 2024b Putta et al. 2024 Hong et al. 2024).
Despite this promising progress, existing models still fall short of realizing such proactive assistant in practice. Their limitations become especially evident in scenarios requiring extremely long-term behavioral monitoring and temporal reasoning. For instance, an all-day assistant should be able to recall whether the user has already taken specific medication hours ago, detect that the same object has been accessed multiple times throughout the day, or notice a warning text previously ignored. Fig. iillustrates detailed case, inspired by scene from the film Memento: the wife asks for an insulin shot three times within a few hours, but the husband, due to short-term memory loss, fails to recognize the repeated requests, potentially leading to serious consequences. In this scenario, existing long-form video models fail to assist during critical moments. They cannot issue timely warnings during the shots and fail respond accurately. On the other hand, even the most advanced online

---

Published as a conference paper at ICLR 2026
Trum 4h
7.5m
> 25min
Ultra-long
Streaming Video
Sammy? Ih's time
shot.
Sanmyy It's line for shot.
It's time for my sholl
User
How many Mines did the womus ouls for a slot? Ressivd moe nech time.
.*...*
Tall me bow long it's been since gove her the previou shot.
Long
Of cousee I will ressidd whenover the onks for
No Response!
No Response!
Exect line bo botweee the fool do! and the preceding $75s.
Video
Online Video LLMs
I (usferstanal Every time the ouls for chat, will remied
2nd time the women niks for shot.
No Response!
No Respanse
Memento
underttand Every lime onls for shot, will sall which occutrence it.
2nd time the
3rd fime the women ooks for
unlerstand Thi. shtt
a
Figure I: Comparison of model behaviors for all-day proactive assistance. Long term and online video models both fail assistant at injection points beyond 25 minutes. Conversely, Memento continuously tracks repeated shots, demonstrating its capability toward serving an all-day proactive assistant. Results for online models and Memento are obtained via supervised fine-tuning (SFT), while long video model outputs are based prompt engineering due architectural limitation.
streaming models struggle with ultra-long durations due their based architectures, which cause visual tokens from each frame to accumulate in memory usage over time. As result, after most a few dozen minutes, the model exceeds GPU memory limits, and cannot recognize that previous requests occurred.
To address the above issues, we propose Memento, a proactive vision-language framework for ultra- long video streams. To handle the long-term memory challenge, introduce Dynamic Memory (DM) mechanism that learns to retain or fuse incoming visual information over time, allowing Memento preserve relevant context while keeping memory usage bounded. In addition, a Query-related Memory Selection (QMS) module that retrieves only the most relevant memory slots during generation, enabling efficient and targeted access across extended video durations. This framework departs from the token-based based paradigm, in which frame-level features are concatenated and multiple positions are supervised jointly. contrast, Memento operates over dynamically updated memory representations, which evolve over time and cannot be aligned to discrete frame steps. As a result, directly applying token-level supervision leads to misaligned inputs and invalid training. To resolve this structural mismatch, we introduce Step-Aware Memory Attention (SAMA), which restricts attention to memory available each step, ensuring temporally consistent and semantically valid learning. While Memento addresses the architectural challenges of proactive interaction with
propose
long-range memory, existing datasets (Chen et al. 2024a; Yao et al.) 2025 Grauman et al. 2022
Yang et al. 2025) offer limited support for training evaluation. Online benchmarks (Chen et al. 2024a| Li et al. 20256 Wu et al. 2024a) include only short-term proactive tasks such as behavior recap based on recent frames, lacking supervision for long-term monitoring. To bridge this gap, construct Memento-54k and MementoBench, covering diverse task types on text, object, and action over video streams up to hours, all requiring long-range, proactive understanding.
Our contributions are summarized as follows:
Framework. For the first time, a framework for proactive interaction over ultra-long video streams, named Memento, is proposed.
Memory modeling. To address the scalable long-term memory challenges, we introduce dynamic memory and query-related selection for selective retention and efficient retrieval.
Training strategy. To enable training compatibility with dynamic memory, we propose step- step-aware memory attention, ensuring stable and effective learning for proactive vision-language modeling.
Dataset and benchmark. We construct Memento-544 and MementoBench, covering diverse long- range proactive tasks, validating the effectiveness of Memento and supporting the development of all-day proactive assistant.
2

---

Published as a conference paper at ICLR 2026
Related Work
Visual Input
Long Form
Proactive
LLaMA- VID (ECCV 2024)
fixed token
TimeSuite (ICLR 2025)
fixed token
MovieChat (CVPR 2024)
fixed memory
MA-LMM (CVPR 2024)
fixed memory
/
Videol.I.M-onil M--onnie (CVPR 2024)
fixed token
X
VideoL (NeurIPS 2024)
dynamic token
LION-FS (CVPR 2025)
dynamic token
Memento
dynamic memory
Table I: Comparison between related methods and the proposed Memento. "Proactive" indicates
whether the model supports interaction without explicit queries.
RELATED WORK
Long-Form Video Understanding. Recent multimodal large language models have demonstrated
strong instruction-followin capabilities in video understanding (Cheng et al. 2024 Zhang et al.
2023 Li et al. 2024b Liu al. 2024b; Wang al. 2025 Zhang al. 2024b), particularly for long-range content. As early approaches based on sparse frame sampling often fail to capture key
clues in long videos (Lin et 2024; Li et al., 2024a| Maaz al. 2024| Ma al. 2024 Zhou et al.)
2024). fixed token-based methods have been introduced via encoding each frame into a fixed number of visual tokens, with compression algorithm for acquiring more frames (Wang al. 2024c Ren et al. 2024] Weng et al. (024). For example, LLaMA- VID (Li et al. (024c)) represents each frame only using two visual tokens, enabling efficient processing of hour-long videos. Beyond token-based compression, fixed memory-based models, including MovieChat (Song et al.) 2024b), Koala (Tan
ct al. 2024). MA-LMM (He et al. 2024), and others (Fan et al.) 2024] Wang et al. 2024b Zhang et al. 2024a), maintain Lixed-length memory bank the visual tokens. They achieve effective line-video compression by aggregating redundant frames with similar features. However, these approaches suffer from increasing inference overhead, limited long-term memory and the inability to proactively interact, making them unsuitable for all-day assistant scenarios.
Online Video LLMs. Online Video LLMS aim to achieve real-time, proactive interaction over streaming inputs, with the ultimate ambition of supporting continuous operation across ultra-long video streams in open-ended scenarios. VideoLLM--nllin (Chen al., 2024a) is the first to enable proactive interaction in video-language modeling by introducing a Streaming EOO objective to decide when to respond or remain silent. However, like other fixed token-based approaches, it requires extracting visual tokens for each incoming video frame, leading to unacceptable growth in memory usage and computational cost. To reduce overhead, subsequent models introduce dynamic token strategies, such as MoF-style (Jacobs al. 1991, Fedus et al. 2022]] Shazeer et al.) 2017 Lepikhin et al. 202T) token routing in VideoLL.M-Mol (Wu et al. 2024b) and LION-FS (Li et al. 2025a), where only subset of tokens are forwarded into deeper layers, and patch-level token dropping in TimeChat-online (Yao et al. 2025), where high redundancy regions are discarded. These methods increase the supported video duration to tens of minutes, but still retain frame token accumulation.
Even the most advanced multimodal models (Wang et al. 2024a Chen et al. 2024c] Gao et al. 2024
Chen et al. 2024b), such as GPT-4o (Achiam et al.) (023) and Gemini 1.5 Pro [Team et al. 2024
2023), struggle to proactively reason over ultra-long streaming video. Unlike prior works, Memento
introduces a dynamic memory design and query-related retrieval, as shown in Table. [ avoiding
token burden and preserving relevant information beyond fixed memory limits. Overall. it paves the way toward reliable, all-day proactive assistants..
3 MEMENTO: A PROACTIVE LLM OVER ULTRA-LONG VIDEO STREAMS
3.1 OVERVIEW
In this section, we introduce our Memento in detail. As shown in Fig. 2 (a), given a streaming video V ffi.ff..... f2...., fr). Memento encodes each frame fi using a Visebased (Radford et al. 2021 encoder. The result U, xux))xxxxxxxx contains global [CLS] token and hp Wy spatial tokens.
Instead of directly projecting i into the language space via an MLP projector as in LLaVA (Liu et al. 2023 2024a), we first process it through the Dynamic Memory (DM). At each step t, the current D, and historical memory M,- are fused according to a Remember-ande -orooeett (R&F) strategy. It decides whether to retain the original information, and produces the updated memory
3

---

Published as a conference paper at ICLR 2026
Streaming
Mernory
Mernery
Forger
Vidco
Forget
Dynamic
Mamory
Cross Attestion
Mem
Mem
Mem
Atem
Mem
Mom
Memoory
FPN
Query-raleted
Memory
Selechion
Q-Mem
(b) Dynamic Memory (DM)
Large Language Model (LLM)
Preactive
R&F Mamory
Q
Cross Atlastion
Q-Mamory
Interaction
[E05
EDSS
Fist
Fmah
Extl
Top
Memento Framework
(c) Query-related Memory Selection (QMS)
Figure 2: Overall architecture of Memento. (a) Memento receives user queries and historical
responses with the current memory state, achieving proactive interaction over ultra-long video streams. (b) Details of the DM mechanism, which mainly utilizes similarity- based retention and aggregation.
(c) Details of the QMS module using query-conditioned gating and masking.
M, (M, M,-1) 1) R&F Memory. Then, for all the queries q (q1- .2.............................. 9n in the past, the current R&F memory M, is filtered by the Query-related Memory Selection (QMS) to retrieve the most relevant subset M M,. The selected memory Mi is fed into the LLM to generate the next-token distributions P, enabling both reactive and proactive responses.
Finally, considering that the fused memory changes across frames but lacks per-frame token structure, we apply Step-Aware Memory Attention (SAMA) to restrict attention to available memory at each time step during training. Thanks to this alignment, the supervision objective from VideoLLM-
online (Chen et al. 2024a can be directly adopted to train the memory-base framework:
C =
N
log logli+1 P p!xxtslll log fj PJJ00)
[39]
(I)
LM Loss
Streaming Loss
where 1, is 1 if the j-th token is language response token, and 0 otherwise. f, is if both (1) the j-th token is the last token in Mi,, and (2) 1y+1 is O. Pj p! is the probability on the j + 1-th text token, output from the large language model head of the j-th token, and |! is the probability for the EOS token.
3.2 DYNAMIC MEMORY
To update R&F memory Mr, aim balance between retaining essential information and fusing redundant content, which may arise in the short term (adjacent frames with little change) or in the long term (repeated scenes or actions), as shown in Fig. (b). To handle both, compute two relevance scores: (1) short-term score 5, based on cosine similarity (Wang et al. 2024d between the current frame and the last memory m,-1 E Rl,,,,,w,,)CC in MM and (2) a long-term
score a, obtained via cross-attention (Vaswaní al. 2017) between ey and all flattened historical
memory tokens M,-1 E NN. (1+hg XmyJxC in Eq. A fixed threshold controls memory update.
(weW,)(M,-1W)
Attn(u,, M,-1) softmax
Vd
σ = ((Attn(v,, M. : (M,) Wo),
where WG, WR. W., and W. are projection matrices. (-.) denotes a summation followed by sigmoid activation (LeCun al. 1998), yielding scalar score a E R.
The R&F gate selects the memory update strategy based on a relevance threshold e. If 5 > the current frame is considered locally redundant and is fused into the last memory token using Eq
score softmax (attaaxAAAAnmmmiiaaaaaaaaa vx)).
score U,
f = m4-1 (1 sum(w)) + w T Ve. Mr = Concat (M -
4

---

Published as a conference paper at ICLR 2026
where score ( is the normalized attention weight in spatial, E IR is fixed scalar update ratio. f is the fused token, and MM denotes the first N₁-1 memory. 6 S while > the frame is semantically aligned with long-term memory content; we thus reuse the same update strategy but compute attention scores by treating M,-1 as queries and o as keys and values, enabling soft updates across all memory slots. Finally, if both 5 S and Ø I c, the frame is considered distinct and directly appended to memory, namely M₁ = Concat(MMe vt).
This gated update mechanism enables Memento to forget redundant content via token fusion, and remember distinct information. Different from voken-based methods, this mechanism could avoid unacceptable growth in memory usage and computational cost. Compared with the fixed-length memory banks, it dynamically expands for novel content. This design maintains compact yet expressive representation across ultra-long video streams.
3.3 QUERY-RELATED MEMORY SELECTION
To reduce memory consumption while preserving response quality, we filter the current R&F memory M, according to user queries q in Fig. (c). Specifically, we transform M, into M. R=Nxx(2thxxx,xXCC and compute cross-attention with tokens Q as keys and values, following Eq. to yield the score R E N. for each memory frame. QMS then applies top-k gating strategy to select the most relevant compact memory M is then passed to the LLM for generation. Our QMS ensures query-awar generation while decreasing the cost of full-memory attention, thereby enabling scalable reasoning over ultra-long temporal sequences.
Tyms .N tokens. M₁ (MM, R. k). The selected
3.4 SWER-AWARE MEMORY ATTENTION
Unlike token-based mod- els with frame-wise cumulation, the memory bank lacks explicit align- ment with video steps. Thus, prior standard train- ing methods in (Chen et al. 2024a; Wu et al. [2024b]
al. 2025a) with causal at-
tention are inapplicable. As shown in Fig. (a), this attention will allow access expired memory. In con- trast, our proposed SAMA in Fig. 3] (b) introduces
Ue
Me
(a) Causal Atlention
(b) Step-Aware Memory Attention (SAMA)
Figure 3: Causal attention vs. SAMA. Causal attention (left) permits access all past tokens, including expired memory. SAMA (right) restricts attention to valid memory, excluding irrelevant tokens.
masking scheme to align with frame- mase visibility.
Specifically, example of input sequence is:
tokens [M, [EOS], M2, 4ı, [Txt]1, Mzs [EOS], , 92. [Txt]2, Mag [Txt]alz- (4)
A binary attention mask A E {0, 1 L is built, where token æ; is allowed to attend to token æz if:
1, I E MM Uqu J 12 Ty * [BOS]
Agj
1. i= x; [EOS]
(5)
0, otherwise
Here, s = step(z;) denotes the video frame index when token F; is added to the sequence. Further- more, reassign correct position ids for each token to ensure that tokens within the same frame share a base offset. This aligns positional encoding with the token visibility defined by the mask.
During inference, maintain the same masking structure that only previous dialog tokens stored as key-value cache (Dao et al. 2023 Ge et al.) 2024), allowing efficient streaming decoding with minimal computation.
5

---

Published as a conference paper at ICLR 2026
Mamento-34K dataset
Mlad KUAA
Nowl DSR
Pleastty (1S00)
per odd
Action
fexty Obijes
Cher, aimi 2014
Ted por
Piceertd 11748
Teaie 10u brode
Counting
Appwar
doudl compleed
1721
Temppral
SJustt (131
152444)
Ordering
Spatial
Disappear
wnle) (1003.1))
Temporal
The les
Ope
mnli (1880
(4 Taequaral Deanging
(6 Taequana
Spettal Abeernal
Duretton
Abnormal
Counting
Objes/ Text
[ive]] 11.00
Nivel 2000
[Mae) (14044N
Changing
Disppeor
city
Objes
(1065
Plemsend (1734.00
[Merwaa) (370.2
(2114.11
Plessend 15336.7
Text
Action
1070
disopeare
appears.
Appeor
Figure 4: Overview of Memento-54k, Left: the 9 task types categorized by spatial vs. temporal, and by modality (text, object, action). Right: example QA instances for each task type.
DATASET AND BENCHMARK: MEMENTO-54K AND MEMENTOBENCH
4.1 MEMENTO-54K DATASET CONSTRUCTION
Video Filtering and Sampling. To support long- dur-tinn, proactive interaction, construct Memento- 54k based on Ego4D (Grauman et al., 2022). We filter all videos to retain those between 5 minutes and 7 hours, ensure long-term context. To reduce sample imbalance, downsample videos from overrepresented scenarios (e.g., cooking), yielding a subset of 4,466 daily-life videos.
Task Annotation. As illustrated in Fig. we define 9 task types spanning spatial and temporal reasoning, where spatial tasks focus on short-term perception (e.g., object presence), and temporal tasks require long-range memory (e.g., repeated actions or text changing). These tasks are designed for three modalities: action, object, and text. Each sample is annotated as streaming QA pair, including question and multiple assistant responses with timestamps. For each modality, we first obtain timestamp-level labels, and then generate QA pairs:
Action. Based on Ego4D timestamp narrations, we prompt GPT-4o to generate QA pairs such as repeated actions of Temporal Counting on the right side of Fig. 4 see Appendix for details.
Object. We extract objects at 2 FPS using ChatReX (Jiang et al. 2024 a category-agnostic detector. QA pairs are then generated via rule- bale- scripts. For example, in temporal duration tasks, we track object appearance and disappearance timestamps identify presence for producing response.
Text. On-screen text is detected by Qwen2-VL at 2 FPS. Text annotations are similar to object, such as temporal changing tasks identify cases where a previously seen full text is later partially disappeared, and once the subset is matched, a response is triggered form QA pair.
Streaming QA Formatting. For each task, up to 9 instances are annotated per video, each fo- cusing on a distinct action, object, or text. Failed or invalid cases are manually corrected. Then, QA pairs are grouped by randomly selecting 1-5 user queries with their timestamped responses to form new streaming samples. This forms the final release of the Memento-54k dataset, and the specific distribution is as shown in Table. 2
Split
Duration
Videos
Samples
Responses
Train
Total
4,426
53.6k
2.5M
5-10 min
10
62
2.6k
10-30 min
10
67
3.6k
Test
30-60 min
15
56
4.8k
>
60 min
13
2.5k
Total
40
198
13.5k
Table 2: Distribution of Memento-54k.
Especially, streaming QA must scan entire videos, making evaluation expensive. Though it contains only 40 videos, the test set covers over 13k responses, which is sufficient for robust evaluation.
4.2 MEMENTOBENCH EVALUATION
To evaluate models under the proactive long-term understanding setting, we identify three essential requirements for this task: temporal alignment, answer quality, and minimal redundancy.

---

Published as a conference paper at ICLR 2026
TimeRecall. TimeRecall is the fraction of ground-truth responses for which the model produces at least one response within a 5-second window, reflecting the ability to anticipate when to respond.
Score. Score measures the generation quality by comparing all the model responses within the above window with ground-truth answers. We use GPT-3.5-turbo- take the maximum among multiple outputs. Scoring details provided in the Appendix.
to assign score from to 10 and
Redundancy. Redundancy captures the extent of unnecessary generation, defined as the proportion of model responses outside the time window in Remele
The most closely related benchmark to ours is Ego4D Narration Stream (Chen et al. 2024a| Lin et al. 2022). which evaluates the temporally align performance generated descriptions with visual events in streaming egocentric videos. However, focuses only on the current narration, overlooking tasks that require long-term past information. In addition, its evaluation relies exact text match, whereas MementoBench supports free-form outputs, enabling more flexible and robust assessment.
Notably, existing online benchmarks (Li et al. 2025b; Wu al. 2024a) such as OVO-Bench, which appear to evaluate proactive interaction, in fact offer an offline-form question and predefined response timestamps during inference. All past video frames before each timestamp are provided, which ideally should be judged by the model. As result, such benchmarks emphasize response accuracy for specified questions, allowing non-proactee non-proactive models to be evaluated under this setting. In contrast, MementoBench compares whether models can proactively interact at the right time with the correct content, enabling more accurate evaluation of the desired capabilities in real-world proactive settings.
5 EXPERIMENTS
5.1
IMPLEMENTATION DETAILS
In this work, we implement our Memento following the VideoLL.M-online framework (Chen et al. 2024a). Unless otherwise stated, we use SigLIP-ViT-L/384 (Zhai et al.) 2023) as the vision encoder, which extracts frame-wise wise features 2 FPS, and set hp W.p = 3. For the LLM module,
LLMMA-3.1-8B 1-8B-Inntruct (Grattafiori et al. 2024). Following (Chen et al. 2024a). we train I epoch for model in the DeepSpeed Zero-2 Rajbhandari et al. (220)) configuration, with LoRA (Hu) et al. (022) to all linear layers the LLM with a rank of 128 and a scaling factor of 256. For our DM module, the top-k ratio Tams 50%. We use Adam W optimizer (Loshchilov & Hutter 2019) with a learning rate of le-4 and cosine decay. All experiments are conducted 4 NVIDIA AT00 GPUs (80GB). Please refer to Appendix for inference details with a dynamic correction strategy.
set the relevance threshold c 0.7, and update ratio 0.2. In the QMS module,
5.2 MAIN RESULTS
We compare our method with VideoLLM-on (Chen et al.) 2024a) using MementoBench. To ensure fairness, we train VideoLL with our Memento- 54k dataset using the same training schedule, denoted as Miteol.LM-ooliness
To assess runtime scalability, Figure ( (right) shows GPU mem- ory usage during streaming video inference. VidnoLLMe quickly accumulates tokens and runs into OOM at about 25 min- utes, with memory peaking at 80.5 GB. In contrast, Memento maintains bounded usage under 45.3 GB across the entire 4-hour streaming videos, demonstrating its advantages for proactive response to ultra-long videos with stable memory and in- terruption. The occasional rises correspond to dense response periods and are reduced afterward temporary variables released.
Memory Usage Time
70
50-
30
Videal1 M-online Memento (Ours)
50
200
250
(minutes)
Figure 5: Memory Usage.
The results on MementoBench are shown in Table 3 The original VideoLLM-online performs poorly across all aspects, with only 6.1% spatial and T1.8% temporal recall, and nearly 0% beyond 25 minutes due to memory overflow. Even after supervised fine-tuning (SFT) on Memento-54k ViddooLLM-online")) average recall only rises to 8.9%, with long-tern recall still at 0.3%. While it reports higher score of 5.32 and lower redundancy of 21.3% this is largely because it triggers
7

---

Published as a conference paper at ICLR 2026
Time 1
Score
Method
Redund. +
Sp.
Temp.
Long (> 25min)
Avg.
Sp.
Temp.
Avg.
Online Video LLMs
VideoLLM-online
6.1%
11.8%
0.1% 0.3%
8.1%
1.55
1.21
1.40
56.4%
VideoLLM-onllnee
7.9%
11.6%
8.9%
5.11
5.68
5.32
21.3%
Ours
Memento*
45.9%
51.3%
35.2%
47.5% %
4.31
4.02
4.22
64.5%
Table 3: Evaluation on MementoBench. Sp. and Temp. denote spatial and temporal task types, where Temp. requires long-term visual reasoning. Long marks responses beyond 25 minutes, for assessing understanding persistence under ultra-long video streams, independent of task type. In particular, VideoL.M-onllin is the only model with available open-source online inference code.
Memory Bank Size vs Time
Memory Schema
TimeRecall 1
Score t
Redund. +
Dynamic (epeno.6)
Dynamic
2407
Sp.
Temp.
Avg.
Sp.
Temp. Avg.
1024
Dynamic (spa+
Memory Size (frames)
Foyed (tPn-
Fixed Memory
Fojd (Lan= 12)
Food [Lame
Len=8
14.8%
22.1% 1%
16.9%
4.61
4.65
4.64
55.5%
Len=32
20.4%
26.4%
22.1%
5.14
5.04
5.12
53.7%
Len=128
28.1%
31.2%
29.0%
4.77
4.74
4.76
52.7%
Dynamic Memory
c=0.6
23.1%
25.5%
23.8%
5.12
5.25
5.16
50.9%
16
c=0.7
38.2%
46.7% % 40.4%
4.36
4.67
4.39
56.2%
c=0.8
43.9%
46.6%
44.7%
4.59
4.05
4.43
61.4%
Table 4: Ablation on memory schema. "Len" indicates the fixed memory bank size. Our dynamic memory consistently yields better recall and offers superior trade-offs others.
50
100
150
200
250
Time (minutes)
Figure 6: Memory Size Comparison.
very few responses, often staying silent when answers are expected. The resulting low recall makes it unsuitable for real-world applications. In contrast, Memento achieves 45.9% spatial, 51.3 51.3% temporal, and 35.2% long-duration recall, while maintaining solid score of 4.22. Although its redundancy increases to 64.5%, given the substantial gain recall (+38.6%), we consider this a worthwhile trade-off, ensuring timely and consistent response is critical in ultra-long online scenarios.
5.3 ABLATION STUDY
We conduct three ablation studies evaluate the core design components of Memento. Our analysis focuses on three aspects: memory mechanism (with 1+2x2 frame tokens, Tams = =000%), frame token configuration (with c=0.7. T:mm=mmm00000 100%) and QMS top-k ratio (with c=0.7, 1+2x2 frame tokens).
Memory Mechanism. To examine the impact of memory structure and hyperparameter on long-term reasoning, compare fixed-length memory banks with dynamic memory mechanism, as shown in Table 4 Increasing the fixed memory size improves recall from 16.9% to 29.0% and slightly reduces redundancy. In comparison, dynamic memory achieves notably higher recall (up to 44.7% at e = 0.8) while maintaining comparable score and redundancy (up to 5.16 and 50.9% at e = 0.6). Notably, for temporal tasks that require long-range memory, recall improves significantly from 31.2% (fixed) to 46.7% at = 0.7. Figure 6 further shows that dynamic memory scales naturally with video length, enabling long-range context retention. However, 0.8 results nearly 10x larger memory than = 0.7 with marginal gain in all the metrics, so we adopt the default c 0.7.
Frame Token Configuration. We futher analyze different frame tokens in Table + 3 x offers better balance, which achieves the highest recall of 68.9%, while maintaining reasonable score of 3.78 and moderate redundancy 66.6%. Fewer tokens achieve too low recall of 40.4%. 1 + 4 increases redundancy without improving recall.
Frame Token| TimeRecall 1 Score Redund. 1
1
40.4%
4.39
56.2% %
1
1 1+3x33 x
68.9 %
3.78
66.6%
144x4
60.9%
3.93
67.2%
Table 5: Ablation on frame tokens.
8

---

Published as a conference paper at ICLR 2026
TimeRecall f
Score
7 Tyms
Redund.
Memory Usage
Sp.
Temp.
Avg.
Sp.
Temp.
Avg.
10%
38.7%
31.5%
33.6%
4.31
4.67
4.33
63.0%
39.53
50%
54.7%
59.5 %
56.1 %
3.96
3.86
3.93
66.7%
45.19 GB
90%
49.0%
52.8%
50.1%
4.15
3.97
4.10
63.7%
53.50 GB
100%
38.2%
46.7%
40.4%
4.36
4.67
4.39
56.2%
55.44 GB
Table 6: Ablation QMS top-k ratio. We exclude textual KV cache in "Memory Usage" reporting.
as dialogue history size varies with response behavior and is independent of Fams-
Time : Ons Os
Time Ob: : 27m 0ds
Time Oh: 35m 05:
Time Oh 44m 38s
Q: When does the event of bolding pulling the rope fosk and?
GR: Whes have lad these consecutive events requence: llokk up'. 'ook around, and 'hold rape'?
action of bolddog rape come up.
pulling been completed.
User
Memonto
:
You'le vo had thane sequentia
eventi: Yook up' (ofter 1.0s)
oction halding the rape bas come
Yook cround (atter
rope"
Figure 7: Qualitative results of Memento ultra-long streaming video. The scene involves rock climbing over 1.5-hour timeline, with three user queries issued at O, O, and 44 minutes, respectively. These queries cover the tasks of spatial appear, temporal disappear and temporal ordering for action.
QMS Top-k Ratio. To assess how QMS filtering affects retrieval relevance, we adjust the top-k se- lection ratio r'mms in the QMS module. As shown in T selecting all memory slots (10m:=100%) results in suboptimal performance: although it achieves the highest score of 4.39 and lowest redun- dancy of 56.2%, its recall is notably lower compared to the best qqus 50% setting by 15.7%. This highlights that overly broad memory access may introduce irrelevant context and distract attention from key visual evidence. Meanwhile, too few slots (r=10%) limits context recall and harms perfor- mance. The 50% configuration strikes the best trade-off across all metrics, demonstrating that QMS effectively prioritizes relevant memory and improves response alignment.
5.4 VISUALIZATION OF MEMENTO
Figure showcases Memento's 's performance 1.5-hour streaming video with temporally distant queries. The model identifies "holding the rope" at 27 minutes in response to an initial query and triggers "pulling the rope completed" 8 minutes later. It also tracks the ordered occurrence of "look up", "look around", and "hold the rope" before issuing a final response. Moreover, it remains proactive across the entire duration, generating correct responses even after 80 minutes, demonstrating its robustness in ultra-long streaming scenarios.
6 CONCLUSION
In this paper, we present Memento, proactive vision-language framework for ultra-long streaming video. introduces dynamic memory, query-related selection and step-aware attention for scalable long-term context modeling and temporally aligned training. Moreover, we construct Memento-54g and MementoBench for training and evaluation. Experiments show that Memento enables effective proactive interaction. Declaration of L.M usage will be discussed in Appendix.

---

Published as a conference paper at ICLR 2026
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, 1lge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang. Wei Ding. Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang, Tianbao Xie, Zesen Cheng, Hang Zhang. Zhibo Yang. Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report. arXiv preprint ar arXiv:2502. 13923, 2025.
Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 18407-18418, 2024a.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye. Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. Science China Information Sciences, 67(12):20010, 2024b.
Zhe Chen, Jiannan Wu, Wenhai Wang. Weijie Su, Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF Conference on Computer 2024c.
Vision and Pattern Recognition, pp.
Zesen Cheng, Sicong Leng, Hang Zhang. Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms. arXiv preprint wrXiv:2406.07470 2024. URL
https:// /ttp//aavvvv///a///vv// org/ abs/ /aabs//200 2206. 07476
Tri Dao, Daniel Haziza, Francisco Massa, and Grigory Sizov. Flash-decoding for long-context inference.
decodinge/dnnngg/// 2023.
Yue Fan, Xiaojian Ma, Rujie Wu, Yuntao Du, Jiaqi Li, Zhi Gao, and Qing Li. [inline- graphee not avail- able: see fulltext/videoagent: A memory-augmentee multimodal agent for video understanding. In Computer Vision - ECCV 2024 - 18th European Conference, Milan, Italy, September 29-October 4, 2024, Proceedings, Part XXII, volume 15080, pp. 75-92, 2024.
William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. J. Mach. Learn. Res., 23:120:1-120:3, 2022.
Zhangwei Gao, Zhe Chen, Erfei Cui, Yiming Ren, Weiyun Wang, Jinguo Zhu, Hao Tian, Shenglong Ye, Junjun He, Xizhou Zhu, et al. Mini-internvl: a flexible-transfer pocket multi-modal model with 5% parameters and 90% performance. Visual Intelligence, 2(1):1-17, 2024.
Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive KV cache compression for llms. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Gird- har, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Car- tillier, Sean Crane. Tien Do,, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Christian Fuegen, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang. Yifei Huang. Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava
10

---

Published as a conference paper at ICLR 2026
Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu. Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Rujjie Tao, Minh Vo. Yuchen Wang, Xindi Wu, Takuma Yagi, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg, Yoichi Sato, Jianbo Shi. Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the World in 3,000 Hours of Egocentric Video. In IEEE/CVF Computer Vision and Pattern Recognition (CVPR), 2022.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning, arXiv preprint arXiv:2501. 2025.
Bo He, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. MA-LMM: memory- -agmented large multimodal model for long-term video understanding. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 13504-13514, 2024.
Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for A multi-agent collaborative framework. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024, 2024.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In Proc. Int. Conf. Learn. Represent., 2022.
Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. Adaptive mixtures of local experts. Neural Comput., 3(1):79-87, 1991.
Qing Jiang, Yuqin Yang. Yuda Xiong, Yihao Chen, Zhaoyang Zeng. Tianhe Ren, Lei Zhang, et al. Chatrex: Taming multimodal 11m for joint perception and understanding. arXiv preprini 1xXiv:241111866 2024.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proc. IEEE, 86(11):2278 2324, 1998.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding. 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li, Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024a.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-interleave:: Tackling multi-image, video, and 3d in large multimodal models. arXiv preprint axXiv:2407.07895 2024b.
Wei Li, Bing Hu, Rui Shao, Leyang Shen, and Liqiang Nie. Lion-fs: Fast & slow video-language thinker as online video assistant. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025a.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In Computer Vision ECCV 2024 18th European Conference, Milan, Ital, September 29-October 4, 2024, Proceedings, Part XLVI, volume 15104, pp. 323-333333300 340,0002255 2024c.
Yifei Li, Junbo Niu, Ziyang Miao, Chumjiang Ge. Yuanhang Zhou, Qihao He, Xiaoyi Dong, Haodong Duan, Shuangrui Ding, Rui Qian, Pan Zhang, Yuhang Zang. Yuhang Cao, Conghui He, and Jiaqi Wang. Ovo-bench: How far is your video-Ilms from rorl-world online video understanding?, 2025b. URL htl.ttts::// patttss///axx wrw./org/aaae abs/ 2501 05510

---

Published as a conference paper at ICLR 2026
Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning. Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FL, USA, November 2024.
12-16, 2024, pp.
Kevin Qinghong Lin, Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Zhongcong Xu, Difei Gao. Kong-Cheng Tu, Wenzhe Zhao, Weijie Kong, Chengfei Cai, Hongfa Wang, Dima Damen, Bernard Ghanem, Wei Liu, and Mike Zheng Shou. Egocentric video-language pretraining. In Advances in Neural Information Processing Systems 35: Annual Conference Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans, LA. USA, November
28 December 9, 2022, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances in Neural Information Processing Systems, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26286-26299 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. nexta-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL https
7/1 alllavavll.. lewa-vl.gittiii wig.t....b.... wo/blog///g/oo / 20242024422222 01-30- ava-next./
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2019.
Fan Ma, Xiaojie Jin, Heng Wang, Yuchen Xian, Jiashi Feng, and Yi Yang. Vista-llama: Reducing hal- lucination in video language models via equal distance to visual tokens. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 13151-13160. IEEE, 2024.
Muhammad Maaz, Hanoona Abdul Rasheed, Salman Khan, and Fahad Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: LLong Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024. pp. 12685, 2024.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul E. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, 2022.
Pranav Putta, Edmund Mills, Naman Garg, Sumeet Motwani, Chelsea Finn, Divyansh Garg, and Rafael Rafailov. Agent q: Advanced reasoning and learning for autonomous ai agents. arXiv preprint ar arXiv:2408.07199 2024.
Rui Qian, Shuangrui Ding, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Dahua Lin, and Jiaqi Wang. Dispider: Enabling video ilms with active real-time interaction via disentangled perception, decision, and reaction. arXiv preprint arXiv:2501.0322 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark. Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 8748 8763, 2021.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: memory optimizations toward training trillion parameter models. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, pp. 20, 2020.
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. In IEEE/CVF Conference Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 14313-14323, 2024.
12

---

Published as a conference paper at ICLR 2026
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoe V. Le, Geoffrey E. Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gates mixture-of-expers ex-eerssss layer. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang. Yan Lu, Jenq-Neng Hwang, and Gaoang Wang. Moviechat: From dense token to sparse memory for long video understanding. In JEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp.
18221-18232, 2024a.
Enxin Song, Wenhao Chai, Tian Ye, Jenq-Neng Hwang, Xi Li, and Gaoang Wang. Moviechat+: Question-aware sparse memory for long video question answering. arXiv preprint arXiv:2404.17176, 2024b.
Reuben Tan, Ximeng Sun, Ping Hu, Jui-Hsien Wang, Hanich Deilamsalchy, Bryan A. Plummer, Bryan C. Russell, and Kate Saenko. Koala: Key frame-conditioned long video-llm. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2024, Seattle, WA, USA, June 16-22, 2024, pp. 13581-13591, 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, Katic Millican, et al. Gemini: a family of highly capable multimodal models. arXiv preprint 122.::232.//18D00 2023.
Gemini Team, Petko Georgiev, Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.055500 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 122...1991 /3971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 5998-6008, 2017.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-languag model's 's perception of the world at any resolution. arXiv preprint arXiv:2409. 12191, 2024a.
Xiao Wang, Qingyi Si. Jianlong Wu, Shiyu Zhu, Li Cao, and Liqiang Nie. Adaretake: Adaptive redundancy reduction to perceive longer for video-language understanding. ArXiv, abs/2503. 12559, 2025. UR. fttpttttt///apptt semant wwcholar. wtw.eeee......oooooooooooooa/ao. 2777066637
Xiaohan Wang. Yuhui Zhang, OrT Zohar, and Serena Yeung-Levy. Videoagent: Long-form video understanding with large language model as agent. In Computer Vision ECCV 2024 18th European Conference, Milan, Italy, September 29- 29-October 4, 2024, Proceedings, Part LXXX. volume 15138, pp. 58-76, 2024b.
Yu Wang, Zeyuan Zhang, Julian McAuley, and Zexue He. Lvchat: Facilitating long video compre- hension. arXiv preprint 2rXi::2402.12079, 2024c.
Yuxuan Wang, Cihang Xie, Yang Liu, and Zilong Zheng. Milm pressure test: Nee- dle in a video haystack, 2024d. URL httpp::/ http:/ggtthh..iiiaageggggeeeeeee noco
WeeddeeinViieeolssssaa
Yuxuan Wang, Cihang Xie, Yang Liu, and Zilong Zheng. Videollamb: Long video understanding with recurrent memory bridges. arxiv, 2024e.
Yuetian Weng. Mingfei Han, Haoyu He, Xiaojun Chang, and Bohan Zhuang. Longylm: Efficient long video understanding via large language models. In Computer Vision ECCV 2024 18th European Conference, Milan, Italy, September 29- October 4, 2024, Proceedings, Part XXXIII. volume 15091, pp. 453 2024.
13

---

Published as a conference paper at ICLR 2026
Cheng- Kuang Wu, Zhi Rui Tam, Chieh- Cen Lin, Yun-Nung Chen, and Hung-yi Lee. Streambench: Towards benchmarking continuous improvement of language agents. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024a.
Shiwei Wu, Joya Chen, Kevin Qinghong Lin, Qimeng Wang. Yan Gao, Qianli Xu, Tong Xu, Yao Hu, Enhong Chen, and Mike Zheng Shou. Videollm-mod: Efficient video-language streaming with mixpure-of-depths vision computation. In Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024b.
Huajian Xin, Z. Z. Ren, Junxiao Song, Zhihong Shao, Wanjia Zhao, Haocheng Wang, Bo Liu, Liyue Zhang, Xuan Lu, Qiushi Du Wenjun Gao, Haowei Zhang, Qihao Zhu, Dejian Yang, Zhibin Gou, Z. F. Wu, Fuli Luo, and Chong Ruan. provveev--- 1.5: Harnessing proof assistant feedback for reinforcement learning and monte-carlo tree search. In The Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025, 2025.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jin Xu, Jingren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang, Keming Lu, Keqin Chen, Kexin Yang, Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang. Ru Peng. Rui Men, Ruize Gao, Runji Lin, Shijie Wang. Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wenbin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zhihao Fan. Qwen2 technical report. arXiv preprint 2rX 2024a.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan. Yang Su, Yichang Zhang, Yu Wan, Yugiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 241.v2241..15115, 2024b.
Jingkang Yang. Shuai Liu, Hongming Guo, Yuhao Dong, Xiamengwei Zhang, Sicheng Zhang, Pengyun Wang, Zitang Zhou, Binzhu Xie, Ziyue Wang. Bei Ouyang, Zhengyu Lin, Marco Cominelli, Zhongang Cai, Yuanhan Zhang, Peiyuan Zhang, Fangzhou Hong, Joerg Widmer, Francesco Gringoli, Lei Yang. Bo Li, and Ziwei Liu. Egolife: Towards egocentric life assistant, 2025. URL http:p:::///xv..////a org/ abs / 22/.//553300333
Linli Yao, Yicheng Li, Yuancheng Wei, Lei Li, Shuhuai Ren, Yuanxin Liu, Kun Ouyang, Lean Wang. Shicheng Li, Sida Li, Lingpeng Kong, Qi Liu, Yuanxing Zhang, and Xu Sun. Timechat-online: 80
URL
org/ abs/ 2504. 17343
Xiangyu Zeng, Kunchang Li, Chenting Wang, Xinhao Li, Tianxiang Jiang, Ziang Yan, Songze Li, Yansong Shi, Zhengrong Yue, Yi Wang, Yali Wang, Yu Qiao, and Limin Wang. Timesuite: Improv- ing MLLMs for long video understanding via grounded tuning. In The Thirteenth International Conference on Learning Representations, 2025.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In /EEE/CVF International Conference on Computer Vision, ICCV 2023, Paris, France, October 1-6, 2023, pp. 11941-11952, 2023.
Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama An instruction audio-visual language model for video understanding. arXiv preprint 2tt.::220600222888. 2023. htLtst/:: org/abs 6.066 02858
Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vst bissh-stream: Memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406. 08085. 2024a.
14

---

Published as a conference paper at ICLR 2026
Yuanhan Zhang. Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruc- tion tuning with synthetic data, 2024b. URL /attp::///xxi......./ http/aaxivvvv...../... wwg./oo///as.... abs/ 2410 02713
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.044 2024.
15


Reviews:
[{"review_id": "TbbHDvwqrR", "reviewer": "Reviewer_BySZ", "summary": "The paper introduces Memento, a vision-language framework for proactive assistance on ultra-long streaming videos. It proposes a Dynamic Memory (DM) mechanism to fuse redundant frames and a Query-related Memory Selection (QMS) module for efficient retrieval. This paper also introduces Memento-54k and MementoBench, a new benchmark derived from Ego4D for training and evaluating long-term proactive tasks.", "questions": "- To solidify SAMA as a key contribution, did the authors attempt to train the DM module without SAMA (i.e., using standard causal attention)? What was the result?\n- In fugure6, why does timerecall drop when increasing r_qms from 50% to above 90%? Does this  suggest the LLM gets \"distracted\" by too much context?\n- How to choose the best model, considering the tradeoff between timerecall, score and redundancy?\n- How is u in eq3 chosen?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses a highly significant and practical problem on long-term online video understanding, which has impact on real-life applications.\n- The paper is well-written with high-quality figures, making it easy to follow.\n- The new benchmark can benefit research in ultra-long streaming videos", "weaknesses": "- The only baseline compared is VideoLLM-online. Figure5 shows this model is fundamentally unequipped for the task, hitting OOM at 25 minutes. To strengthen the experiments, the model needs to be compared with other sota models, such as VideoLLM-MoD and LION-FS. To provide a comprehensive evaluation, it would be helpful to validate the proposed method on the Ego4D Narration task that is the same as used in VideoLLM-online.\n- Some evaluation metrics are ambiguous. (i) Score is a numeric value from 1 to 10 assigned by gpt3.5. It's unclear if it is a calibrated metric and how good gpt3.5's judegement is. (ii) Redundancy is \"the proportion of model responses outside the time window in TimRecall\". It's unclear if it's a delay of responses or hallucinations.\n- The Dynamic Memory mechanism appears very sensitive to hyperparameters. For example, Figure 6 and Table 4 show that changing $\\epsilon$ from 0.7 to 0.8 causes the memory bank to grow 10x larger for only a marginal performance gain. This suggests that the model needs to be carefully tuned when applying to new domains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t7vhBPYyuN", "reviewer": "Reviewer_kAWu", "summary": "The paper presents a vision-language framework that can handle proactive-understanding in a long (multi-hour) streaming video setting. It addressed the memory exploision issues in prior video-LLMs and related work videoLLM-online. \n\nThe method mainly provides three modules as the technical contributions: \n* It used a dynamic memory module to handle redundant frames, using the heuristic attention based mechnism to compare the similarity of new frames to history frames. \n* It introduce a query-related memory selection mechanism to retrieve key frame memory related to current query. \n* To handle the frame alignment with the presence of two modules above, it introduces a step-aware memory attention mask to prevent access to expired frames during training. \n\nTo evaluate, the paper presents a Memento-54K, a benchmark dataset built on top of existing ego4D dataset, and they generate QA pairs categorized into action, object and text modalities. The ablations confirm the effect of the method. Compared to prior seminar work videoLLM-online (trained similarly), the proposed method can effectively handle multi-hour long inference, while videoLLM-online will be run into OOM after roughly 20 mins in the same setting.", "questions": "Mostly the first question I left in the weakness. I think even some simple experiments to evaluate the current model with existing trained checkpoints on a few small cross-domain datasets will be good to give some clues. I would expect it may degenerate, but I am interested to see how reliable it is to handle things if query is getting out of distributions.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall I found this paper is well-motivated, technically sound, and validated well with the experiments. \n* The proposed three-modules handles the redundant frames, query-memory attention, while still being frame-aligned as videoLLM-online. The three modules compound to each other well. I believe such mechanism can also be extended to other long-video understanding problems. \n* The performance on long-video understanding is validated well, on the new proposed dataset, compared in an ablation setting and with prior work. The method in particular highlights its strong recall, which is an impressive improvement.", "weaknesses": "* It is not entirely clear how the method will perform in a non-templated QA scenario (if not categorized in the action, object, text query). I am particular interested in that how will the query-related memory selection module generalized in non-templated scenario. It may or may not generalize depends on the complex of the query, which may provide some clues whether current simple cross-attention heuristics is sufficient to handle a more complex setting. \n\n* A small writing issue, the paper use big \"M\" and small \"m\" to describe the accumulated historical attentions frames and individual frames if I understand correctly. It will be more easily understood if can be clarified in the writing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DL7d4juGfw", "reviewer": "Reviewer_xktG", "summary": "This paper introduces Memento, a framework that aims to endow video-language models with persistent long-term memory for streaming video understanding. Instead of treating each clip independently, Memento incrementally updates a memory bank of “key events” or “summaries,” enabling the model to maintain temporal continuity and answer long-term queries across extended video streams.", "questions": "* Why are results shown only on VideoLLM-Online? Have you tested on other long-video or streaming benchmarks such as StreamingBench, LVBench, or EgoSchema?\n* Does Memento improve performance in non-streaming (offline) scenarios as well, or is its advantage restricted to streaming conditions?\n* The paper emphasizes long-term efficiency — could you provide actual latency, memory usage, or computational cost metrics to substantiate this claim?\n* How does Memento compare to other recent memory-based systems under similar settings?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "* The paper tackles an important and timely problem, i.e., enabling continuous, long-duration video understanding rather than static or episodic reasoning.\n* The conceptual framing (“all-day memory”) is inspiring and forward-looking.\n* The hierarchical structure separating short-term vs. long-term memory mirrors cognitive and neuroscience-inspired models, offering conceptual richness.\n* Memento appears flexible enough to plug into existing Video-LLMs, potentially generalizing beyond the specific tested backbone (LLaMA 3.1).\n* The paper presents detailed ablation studies.", "weaknesses": "* It’s not fully evident why such a complex hierarchical design is necessary. \n* The overall design reads as over-engineered for the stated goal. Similar effects might be achievable with simpler keyframe/event summarization or query-conditioned caching.\n* Experiments are restricted mainly to OVO, with no results on other benchmarks (e.g., StreamingBench, EgoSchema, LongVideoBench, etc.), making the generality claims weak.\n* Comparisons are missing with more baselines (for instance non-streaming methods).\n* No offline results or cross-dataset evaluations are provided. It’s unclear if Memento benefits standard (non-streaming) settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BlkoU6JrK0", "reviewer": "Reviewer_Rvrh", "summary": "The paper proposes Memento-54K and MementoBench, a dataset and benchmark for proactive long egocentric video understanding. The paper also presents Memento, a training-based framework for streaming proactive video understanding with multimodal large language models. Memento enables ultra long video understanding by devising a dynamic memory mechanism and an efficient memory retrieval method. Qualitative and quantitative evaluations show that Memento greatly outperforms existing offline or online multimodal large language models in terms of proactive video understanding tasks.", "questions": "- Are the quality of the LLM-generated content for the dataset and benchmark verified by humans? Given that LLM-generated content can still be unreliable, it is important and common practice to manually inspect and verify the QA-pairs or detected objects/text. It would also be helpful to provide some examples in the appendix and discuss some failure cases, if any.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is mostly well written, clear and easy to understand. The problem is well-motivated with clear motivating examples in the beginning demonstrating the failure modes of existing methods.\n- The paper presents a large amount of work, including a framework, a dataset and a benchmark, which are all good contributions to the field.\n- The paper presents a training-based framework. Despite being more expensive, training-based methods have more potential than training-free methods, and are more technically challenging to get to work. The paper presents many details about their model training strategy such as the step-aware memory attention (SAMA) that could be used in other streaming understanding applications.\n- The paper includes detailed ablation studies that provides useful insights for configuration the model and training.", "weaknesses": "- The paper only presents a very limited number of baselines in the experiments, mostly just VideoLLM-online, which has been shown to be a fairly weak model even when compared to others in the online setting. I would encourage the authors to add more simple baselines with existing offline video models. More specifically, if would be helpful to show quantitative results for the \"long video LLMs\" row in figure 1 and further demonstrate the inefficacy of existing models regardless of prompting strategies.\n- Although the paper devises a dynamic memory mechanism to prevent undesirable accumulation of visual tokens over time, it still requires repeated filtering of the memory for each user query (line 182). Hence it is a valid concern that the model might not scale very well with a much bigger number of queries (which is completely conceivable in real-world applications). I hope the authors to provide more discussion on this limitation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)