$\infty$-Video: A Training-Free Approach to Long Video Understanding via Continuous-Time Memory Consolidation
Authors: Saul Santos
António Farinhas
Daniel C McNamee
Andre Martins
Venue: ICML 2025 poster (2025)
Keywords: Long context
Video understanding
LLM
Continuous attention
VLLM
Primary area: deep_learning->attention_mechanisms

Abstract: Current video-language models struggle with long-video understanding due to limited context lengths and reliance on sparse frame subsampling, which often leads to information loss. In this paper, we introduce $\infty$-Video, which is able to process arbitrarily long videos through a continuous-time long-term memory (LTM) consolidation mechanism. Our framework augments video Q-formers by making them able to process unbounded video contexts efficiently and without requiring additional training. Through continuous attention, our approach dynamically allocates higher granularity to the most relevant video segments, forming "sticky" memories which evolve over time. Experiments with Video-LLaMA and VideoChat2 demonstrate improved performance in video question-answering tasks, showcasing the potential of continuous-time LTM mechanisms to enable scalable and training-free comprehension of long videos.

00- VO-VIDE:: via Continuous-Time Memory Consolidation
A Training-Free Approach to Long Video Understanding
Saul Santos 12 António Farinhas' 12 Daniel C. McNamee 3 André F. T. Martins 2
Abstract
dation processes adaptively integrating important episodic events into long-term memory (McGaugh, 2013; Cowan et al., 2021). How
Current video-language models struggle with long-video understanding due to limited context lengths and reliance on sparse frame subsampling. often leading to information loss. This paper in- troduces oc- VIDEO, which can process arbitrarily long videos through a continuous-time long-term memory (LTM) consolidation mechanism. Our framework augments video Q-formers by allow- ing them to process unbounded video contexts ef- ficiently and without requiring additional training. Through continuous attention, our approach dy- namically allocates higher granularity to the most relevant video segments, forming "sticky" mem- ories that evolve over time. Experiments with Video-LLaMe and VideoChat2 demonstrate im- proved performance in video question-answering tasks, showcasing the potential of continuous- time LTM mechanisms to enable scalable and training-free free comprehension of long videos.
ensure models able fully un- derstand and grasp information from urbilrarily long videos while they process them without losing critical details?
Transformers offer significant potential for extracting spatio- temporal features from videos (Zhang et al., 2023b; Li et al., 2024). While recent video-language models have prioritized simpler methods such as projection layers (Li et al., 2023d; Liu et al., 2023; Li et al., 2023b; Liu al., 2024c; Ye et al., 2024) and temporal pooling (Luo et al., 2023; Maaz et al., 2024) for the sake of efficiency and scalability, these ap- proaches often sacrifice transformer's representational depth. Additionally, training video- lideo-language models presents sig- nificant difficulties, whereas traditional approaches mainly focus on scaling model parameters (Liu al., 2024c; Cheng et al., 2024; Li et al., 2024), which requires substantial computational resources. A recent alternative explored the usage of additional computation during inference (Zhang et al., 2023a; Wang et al., 2024a;c). However, these methods assume that the spatio-temporal module is intrinsic to the LLM, which limits their ability to effectively specialize in capturing spatio-temporal content. Furthermore, these meth- ods often rely on subsampling and are designed to process the entire video whenever they need to answer a question.
1. Introduction
Multimodal large language models have driven progress in video-language tasks through the integration of pretrained visual encoders with powerful text-based models (Li et al., 2023b; Zhang et al., 2023b; Cheng et al., 2024; Li et al., 2024). However, current video models are constrained by short context lengths (Li et al., 2023b; Maaz et al., 2024; Liu et al., 2023) and often rely on sparse frame subsampling for longer sequences, which limits their ability fully pro- cess and understand long videos. This contrasts with the high-capacity persistence of human memory (Brady al., 2008) and the cognitive principles by which humans store information over long timescales, which involve consoli-
In this paper, we take an alternative approach inspired by hu- man cognition, where memory consolidation processes en- able the retention and efficient handling of long-term depen- dencies (Frankland & Bontempi, 2005; Preston & Eichen- baum, 2013: Song et al., 2023; 2024; Balazevic al., 2024). Namely, we develop new framework with continuous- time visual memory representations. Our framework adapts the co-former architecture previously developed for textual data (Martins et al., 2020; 2022b), which we lever- age to extend the capabilities of pre-trained short-context multimodal LLMs, making them able to process unbounded video contexts in a training- free manner. Shifting from a discrete a continuous attention framework parallels the recent evolution in theories of human working memory diated by prefrontal cortex- ffrte the discrete "slot-based" model to the continuous "shared resource" approach (Ma et al., 2014). Consequently, our method aims to cultivate
'Instituto de Telecomunicações "Instituto Superior Técnico, Universidade de Lisboa 'Champalimaud Research "ELLIS Unit Lisbon *Unbabel. Correspondence to: Saul Santos <saul.r.santos @ becnico.ullsboa.pt..
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

D0- VG-VIDE:: A Training-Frey Free Approach to Long Video Understanding via Continuous-Timm Time Memory Consolidation
ba Mixale
man charedtrrrrrr
Sticky the somes ih mac kio anryy Srowe Ni monigg lorigrusid
Ulice
Misual Encoodr
Visual Encoder
Uvilorn Ansiiuat in foe cores thy wush deeratee in was cauly cndy mooiin nal arattin ui aasm with aurum
Chunii
Chunk
Adtrimaa
sran with Tor
Q-formor
Lsez le the fine perkermnnee indoces setdame?
Slick Amblant tiin Aran profermnne Hany plocr ow
Video C-fonne
former
STM
desctiation the beattns
Taaserr yolociie seara dline euth dus ooer adtur seow
Merorr LTM
Profector
Protector
Desc What the hur gence music as dueen video?
Thr ras by accaneed he awd or deemationeed mork.
Query tokens
Unilor Ansisteet freed sikos that prinenly nect cxmic atheugh a ccckecly esevicned
Shared weights
Frazen
Question
LCM
Stidky Amiddant: in the but wore dirmas dimncter pile fhr knyloand
Figure 1. (Left) Overview of 0o- VI-EID (our approach) using Vidco LLaMA (Zhang et al., 2023b, gray arrows), which uses an additional spatial Q-former module, and VideoChat2 (Li et al., 2024, black arrows). We split the video into frame chunks and apply these models to each chunk. The Video Q-former module combines weighted average of the STM, which is the attention for an individual chunk, with a continuous LTM that takes into account previous chunks. The outputs of the Video Q-Former are projected and then averaged. The LLM takes as input visual tokens, generated by our modified video Q-former, alongside the corresponding question to obtain the answer. (Right) Examples of co- Video LLaMA answers, equipped with our LTM, with uniform sampling and sticky memories for short and ultra-long videos. Italicized corresponds to the correct answer, while underlined corresponds to wrong answer or hallucination.
similar insights within the realm of episodic memory pro- cessing, where dynamic handling of memory is essential
long-video datasets.
We validate our approach using the Video-LLaMA (Zhang process and organize information as it comes, with only one pass over the video. Our main contributions are:
This leads to oo- VIDEO models (Fig. 1), which are able to
et al., 2023b) and VideoChat2 (Li et al., 2024) models by processing a stream of video frames with a single pass, where the STM considers one chunk at the time, and the LTM maintains global information from past chunks. Our proposed model shows improved performance video question-answeree tasks when using the LTM and competitive results with other training-fres free models.
We equip the current attention mechanism of video Q- formers (short-term memory. STM) with a continuous- time LTM that consolidates video information by dynam- ically allocating higher granularity to the most relevant parts of video.
2. Backkround
We develop new continuous-tin attention mechanism which is more powerful than the Gaussian model of Mar- tins et al. (2022b) by considering the Gibbs density based on the continuous query-key similarity function.
2.1. Discrete Attention
Attention mechanisms (Bahdanau et al., 2015) act as a mem- ory component modern neural networks, enabling them to dynamically focus on key parts of the input and capture long- range dependencies, improving performance across various tasks (Vaswani et al., 2017; Dosovitskiy et al., 2021).
We show that architectures with spatio-temporal feature extractors designed for short videos can generalize to long- video understanding in simple, training-fres manner without requiring task-specific fine-tuning or training
Consider two sequences X E L and Y E I Rxe, where L and R are the sequence lengths and is the embedding size. vanilla attention mechanism in transformer works
TThe code is made available in https::///// httpa:////thbb..////// com/ deep-spi spin/ inffnnitevei
2

---

VI-VIDE: A Training-Free Approach to Long Video Understanding via Continuous- Time Memory Consolidation
as follows. First, we obtain queries (Q), keys (K), and val-
discrete attention. The context is then computed as the expected value of the values v(t) Tav())
ues (V) by linearly projecting X and Y for each attention head/ h:
Z = E (()]-
(6)
Qa =YW K² = -X V XW
(1)
co-former. In a transformer with discrete attention, han- dling a long context (large L) becomes impractical due to excessive memory demands. The oc-former (Martins et al., 2022b) overeomes this limitation by means of an un- bounded LTM leveraging continuous attention ($2.2). It allows for unbounded context without increasing memory usage by trading off the number of basis functions that fit into memory with the granularity of their representations. It achieves this by sampling points within the [o, 1] interval, either uniformly or based on prior attention, at which (+t) is evaluated. This process, as we will see in $3, can be seen as the memory consolidation step, allowing new informa- tion from the short-term memory xtt) to be incorporated by scaling it down with forgetting factor т. The past context is associated with positions in [0, r]. while the new context is associated in (r, 1], followed by ridge regression over the new a+l) and computation of the output context as Eq. 6.
where W Rexd W E Rex, and W E Rexd are head-specific learnable projection matrices, d = e/ [h] and [h| is the number of attention heads. For each head, the context representation Zh E Lxd is computed as:
Qa (K((KT)T) Vd
Z softmax
V
The outputs from all heads are then concatenated to obtain the final context representation Z E R__xe
Z = Z z
z h| Wz, 2
where Wz E Rexe is another learnable projection matrix.
2.2. Continuous Attention
Instead of splitting the input object into finite set of pieces (e.g., tokens in text or pixels in images), continuous attention mechanisms (Martins et al., 2020) assume an underlying continuous domain, suitable for arbitrarily long temporal signals, such as audio or video data. This is done by replac- ing the attention probability mass function by probability density function (PDF) over a continuous signal.
2.3. Video Q-former
A video Q-former (Zhang et al., 2023b) is a specialized vari- ant of a transformer architecture designed to enable LLMs to process and understand video content. Developed ini- tially to capture spatial features in images (Li et al., 2023a), its primary function is to map L x P spatial embeddings, where P can be the number of patch vectors from a visual encoder, to R spatio-temporal video representations. It oper- ates by stacking layers that first apply self-attention between R learned queries. followed by cross- self-attention output and the I. P spatial embeddings, leading to Rt spatio-temporal representations.
In continuous attention, the input is assumed to be a con- tinuous signal (l). Although video data is "continuous- time" in nature. it comes as a stream of L discrete frames ..... x E RLxe, and therefore it is necessary to convert this sequence into a smooth continuous signal. This can be done by expressing the continuous signal (x)) Re as linear combination of N basis functions (t) R.V
X = [
between this
3. Unbounded Memory Video Q-former
(t) = B (t),
(4)
In this section, we describe our approach to endow video models with a continuous-time memory mechanism. We adapt existing video Q-former models (Zhang et al., 2023b; Li et al., 2024) by first splitting the full sequence of frames into chunks and then processing each chunk individually. Each chunk includes discrete STM, which is the already present cross-attention in that chunk. However, building the LTM assumes continuity within the embeddings, which is not the case since each frame contains P distinct embed- dings. To address this, we perform average pooling over the P embeddings, resulting in discrete sequence X E Maxe where M denotes the number of frames in the chunk. We
where B E gxx is coefficient matrix. For compression, it is appealing to use smaller number of basis functions than frames, N & L. B can be computed with multivariate ridge regression (Brown& & Zidek, 1980), where we set the domain of the continuous-time signal to the unit interval [0, 1). Frames are associated with time instants in this unit interval, 4 S l2 S ... S Lt. with each te [0, 1), and we set our design matrix F = [=b(tt)..... (tz)) E NL The coefficients B are computed such that axtt) = x; for each frame é E {l....,L) L}. with x= > 0, leading to:
= (FF -
(5)
FFrr instance, in Martins et al. (2022b), the PDF is modeled as Gaussian distribution A(t:; μt, a²), where the parameters and a² are learned by neural network. In $3, we propose a different, more powerful strategy.
The final step involves attending to (t). In this approach, a PDF pt)) replaces the probability mass function of the
3

---

Ve-VIDE: A Training-Free Approach to Long Video Understanding via Continuous- Time Memory Consolidation
introduce global and dynamic LTM, which works with modified, more powerful version of the continuous attention mechanism in $2.2. The LTM update enables increased granularity in memory regions with higher cumulative atten- tion density.
Past (t)
new
contraction
concatenation
sampling
3.1. Long- Term Memory
The first step in building our continuous LTM is to project the long-ferm continuous input x=({) = B {b))), leading to the continuous keys (t) IRd and values (t) E R. with the same projection matrices (I) of the STM. This allows us to perform attention over the same embedding space the vanilla video Q-former as:
a=t)
regression
kh (W T xt) = (W B (t),
(7)
o0(1) (W T xx1) - (W B (1).
(8)
Figure 2. Proposed Memory Consolidation Mechanism.
We compute the query Q* = |9
..... إل .
Y w as in
and build the matrix X past [2T E RTxx, Following this, we concatenate the current step context Xnew E M xd with the previous context X E Tx resulting in the combined sequence:
(1). For each query q: E R2, we compute the continuous query-key similarity s² (t) for each head and query as
(t) q: k(t)= = q. (WR) BB" (t)),
(9)
X =
X Xpstt- Xnow] T E RRT+M)XE,
(14)
and compute Gibbs PDF
exp(s (1 I exp(3 di
With the new and previous chunk contexts, we compute B, as described in Eq. 5, where the continuous signal is approximated using linear combination of rectangular functions. To achieve this, first adjust the contribution of the previous context using the factor т. Specitically, we associate X past with positions in the interval [0, r] and Xnow with positions in (T, 1]. This process yields matrix G E R(M+L)X During each step, the previous context is contracted by a factor of T, which regulates the extent of long-term serg-termmnn memory used for attention and induces gradual "forgetting" process, After the computation of B, continu- ous attention is performed described in $3.1.
(10)
where the integral is approximated with the trapezoidal rule. Given the value function v² (t). we compute the attention- specific representation vectors described in (6):
n = E,, 'o
(f) (t)
( T T (AOON()) (11)
Finally, we obtain the LTM representation ZLTM by concate- nating the context heads and projecting them as in (3).
3.2. Continuous-Time Memory Consolidation
3.3. Sticky Memories
As we continue processing chunks, the LTM is progres- sively updated, as illustrated in Fig. 2. This is done by first sampling T' locations within the interval [0, 1] and then evaluating the continuous signal (t), or current LTM, at be performed either will
When extending the LTM, the signal is evaluated T loca- tions within [0. 1]. as described in $3.2. While these loca- tions can be uniformly distributed, allocating more "memory space" to regions of higher relevance ensures that critical information is prioritized in line with continuous resource allocation conceptualizations human memory (Ma et al., 2014). This selective allocation compresses the signal into an efficient representation whereby essential video details are retained, and less significant ones are compressed discarded. The recurrent and adaptive nature of this process is reminiscent of memory consolidation and reconsolidation brain mechanisms for relevance-basee long-term memory transformation (Hardt et al., 2010; Dudai et al., 2015).
these sampled points. The sampling
uniformly [0, or based on prior attendance, explain in detail in $3.3. The next step involves concate- nating this LTM context with the new one coming from the current chunk. To do this, we first "contract" the LTM as
x' (t) = xt//r) -B (t(TT)
(12)
We then compute æ1) the T locations 0 5 l...., lT ST:
x; B B (ts/Tt (T),
(13) To address this, we propose selecting the T locations based

---

DI-VIDE: A Training-Free Approach to Long Video Understanding via Contimuous- Memory Consolidation
on the relevance of the signal in each region as done by Martins et al. (2022b). This process begins by construct- ing histogram of the previous attention across intervals. Specifically, the signal is divided into D linearly spaced bins, {di,..., dp}. The probability assigned to each bin, p(d,) for j E [D]. is computed by integrating Eq. 10 over each bin interval using the trapezoidal rule:
tion answering ($4.2) and open-ended generation ($4.3).
4.1. Implementation Details
The Video-LLaMA (Zhang et al., 2023b) architecture that we adapt here with our LTM module was initially designed for short videos and employs dual Q-Former architec- ture (Li et al., 2023a) for spatial and another for tem- poral feature extraction. We use the Video-LLaMA2-7B finetuned model, leveraging EVA-CLIP's ViT-G/14 (Fang ct al., 2022) as visual encoder and Vicuna 7B (Chiang et al., 2023) as our LLM.
H
R
p(d,) x
ΣΣ (1). h=1
(15)
Finally, T locations are sampled based on the resulting dis- tribution, followed by the L.TM update of $3.2. In our model, this sampling process is analogous to the phenomenon of non-local discontiguous "replay" in the brain whereby past experiences are reactivated for the purposes of consolidation (Carr et al., 2011; McNamee, 2024).
We also adapt VideoChat2 (Li et al., 2024), a stronger short- video model equipped with a single video Q-Former and trained on extended instruction data. We use UMT-L (Li et al., 2023c), which captures both spatial and temporal dependencies but requires more memory. Thus, we use chunks with fewer frames. Finally, we follow Jiang et al. (2023) and use Mistral-7B (Jiang et al., 2023). Additional implementation details can be found in App. A.
3.4. Model Architecture
We define the output context for the cross-attention layers of the video Q-former as a weighted sum of two components: the "local" vanilla video Q-former output context ZSTM E I Rxxd which corresponds to the already present attention over the current chunk, and the "global" LTM context ZLTM. which takes into account information from previous chunks as described in $3.2. The overall context is computed as:
In all our experiments, we approximate the integrals of Eqs. 10 and 11 using the trapezoidal rule with 1000 sam- pling points. For VIDEO with Video-LLaMA, we use 8 chunks of 256 frames with 1024 basis functions, except in the case of NeXT-QA (Xiao et al., 2021), where the total number of available frames reduced. For co- VIDEO with VideoChat2, we use 8 chunks of 16 frames with N 256 basis functions. We experiment with several values of a in Eq. 16. Full hyperparameters can be seen in App. A.2.
= a ZSTM + (1 - a) ZLTM
(16)
where is weighting factor that balances the contribu- tion of short-term and long-term memories. The video Q- formers considered in this work were trained with the R tokens (or a subset) concatenated with the prompt tokens. In our approach, where the video context is divided into frame chunks processed by the long-term mechanism, it is necessary to aggregate information from all the C' chunks, in RCxi, into a fixed set of R tokens. To achieve this, we run the modified Video-LLaMa (Zhang et al., 2023b) and VideoChat2 (Li et al., 2024) models for each chunk and compute a running average of the embeddings as we process each chunk. The current video token embedding E. E Rxd, defined as E.- WooizZz Ze. updated incre- mentally, enabling the model handle arbitrarily long con- texts without storing all embeddings of chunks in memory. The updated embedding is calculated as:
4.2. Multiple-Choice Question Answering
4.2.1. COMPARISON WITH OTHER TRAINING-FREE METHODS
We consider systems of three kinds: (1) training-free ap- proaches leveraging GPT-4 backbone (Zhang et al., 2023a; Wang et al., 2024a:c); (2) models most similar to ours, which share the same underlying architecture, such as Video- LLaMA (Zhang et al., 2023b) and its training-free variants like MovieChat (Song et al., 2023) and MovicChat+ (Song et al., 2024); and (3) VideoChat2 (Li et al., 2024). For (2) and (3), we test VIDEO variants without LTM (corre- sponding to = 1.0) and those using uniform sampling and sticky memories with a = 0.9.
E. C-=C-1B.11 + E.,
C
1
E. = C
(17)
NeXT-QA. We evaluate our models in NeXT-QQA (Xiao et al., 2021), a dataset with questions and 5 multiple choice options about short videos with average duration of 44 seconds. As co Videidoo LLaMA can process a higher number of frames, we anticipate that using all the video information may lead to improvement over sub-sampling. For co- Video
Upon reaching the final chunk, the video token embeddings E are fed to the LLM that generates answer. The full architecture diagram is shown in Fig. 1.
4. Experiments
In this section, we evaluate our proposed method on video question answering tasks, including multiple choice ques-
/ttg:///g//gg/gaacceeaao cww.eeeoo///O---// NAPMO-NNPP-G. Video- LLaMA 2-7B-P -- Pinetured
5

---

Do-VIDE: A Training-Free Approach to Long Video Understanding via Continuous- Memory Consolidation
Table Evaluation on Multiple Choice Datasets. Evaluation ac- curacies on NeXT-QA (NeXT) (Xiao et al., 2021) and Egoschema subset (Ego) (Mangalam et al., 2023). denotes results obtained by running the models on both datasets. indicates models run Egoschema but not on NeXT-QA, o highlights models trained on NEXT-QA. The remaining results from Song al. (2024) or Wang et al. (2024c). We bold the best-perg models for cach category.
Table 2. VideoMME results. Baseline results are taken from (Fu et al., 2024). We bold the best performing models.
Method
LLM
#Frames Medium Long Avg
ST-LLM
Vicuna-78
64
36.8
31.1
37.9
Video-ILLaVA
Vicuna-7B
8
38.0
36.2
39.9
SharcGPT4Video 8B
16
36.3
35.0
39,9
Chat-UniVi-vl.55
Vicuna-BB
64
40.3
35.8
40.6
Qwen-VL-Chat
Qwen-7B
4
38.7
37.8
41.1
VideoChat2
Mistral-7h 7B
32
37.9
38.0
42.1
Method
LLM
#Frames
NeXT
Ego
0c- Co-VidoChha22 (no LTM)
Mistral-7F
128
39.6
38.8
42.3
Based on Proprictary LLMs
oc- VideoChat2 (uniform)
Mistral-7B
128
40.0
38.8
42.4
LLvvi (Zhang et al., 2023a)
GPT-4
67.7
61.2
oc- V--iidoChht2 (sticky)
Mistral-7B
128
40.2
38.9
42.4
Videodgent (Wang et al., 2024a)
GPT-4
71.3
60.2
VideoTree (Wang et ul., 2024c)
GPT-4
75.6
66.2
Video -aased Models
Video LLaMA* (Zhang et al., 2023b)
Vicuna-7B
32
30.7
20.2
MovieChatf (Song et al., 2023)
Vicuna-7B
2048
34.4
41.6
VideoChat2, though the improvements are less pronounced.
MovieChatti (Song et al., 2024)
vc-Video LLaMA LLMMA (No LTMY"
Vicuna-7B
Vicuna-7B
all/2048
2048
37.6
35.2
40.8
37.4
We attribute this to the fact that VideoChat2 is inherently a stronger model, having been trained on a larger and more recent datasets, offering less room for improvement com- pared to the Video-LLaMA oo-variants. However, gains are observed for both uniform sampling and sticky memories, with the latter showing superior performance. Moreover, de- spite having significantly fewer parameters, oc- sc-VideoChat2 demonstrates competitive results against proprietary LLMs based on ChatGPT-4.
xo- Video LLaMA (Uniform))
Vicuna-7B
all/2048
37.5
42.6
oc- V--Vidoo LLaMA (Sticky)*
Vicuna-7B
all/2048
41.1
46.8
OeleoChar2-Baxed Models
*o (Li et al.. 2024)
Mistral-7B
78.7
64.2
so-VideoCha22 (No L
Mistral-7B
128
78.1
64.6
VideoChat2 (Uniform)*o
Mistral 7B
128
78.1
64.4
oc- VVieeoChat2 (Sticky)*o
Mistral-7B
128
78.1
64,8
LLaMA, we use all available frames in the videos, which we then split into chunks of up to 256 frames. In contrast, baseline models like Video-LLaMA (Zhang et al., 2023b) are limited to processing only 32 frames, while VideoChat2 achieves optimal performance for 16 frames, as shown in Li et al. (2024).
4.2.2. EVALUATION ON VERY LONG VIDEOS
To emphasize the effectiveness of our approach on extended video content, we also present results on Video-MME (Fu et al., 2024), which features diverse collection of lengthy videos, ranging up to I hour in duration. We compare our method against baseline models of similar size such as ST- LLM (Liu et al., 2024b), Video-LLaVA (Liu et al., 2023), ShareGPT4Video 8B (Chen et al., 2024a), Chat-UniVi-vl.. (Jin et al., 2023) and Qwen-VL-Chat (Bai et al., 2023) as well as our xo- VIDEO variants with a 1 and the base architecture VideoChat2 (Li et al., 2024).
As shown in Tab. I, oo- Video LLaMA with sticky memories outperforms other approaches. This includes MovieChat+, which employs heuristic to merge adjacent frames, fed to the video's 's Q-former cross-attention with question knowl- edge. In contrast, our model encounters the question only within the LLM prompt. Surprisingly, our method using uniform sampling of the continuous signal performs slightly worse than oo-Vides Video LLaMA without the LTM (a = 1). For the VideoChat2 variants, we observe no significant per- formance increase. We attribute this to the in-domain nature of the evaluation, where the original model is already highly optimized for the given tasks.
In Tab. 2, we show the results for VideoMME. Al- though Video-I.LaMA performs well on earlier datasets, its description-focused training dataset makes the performance as good as random guessing. Including it in the evaluation would detract from more relevant models. However, for the VideoChat2 category, sticky memories outperform others, followed by uniform sampling.
Egoschema. We evaluate the training-free question- answering performance of our models on EgoSchema (Man- galam et al., 2023), a medium-length benchmark for ego- centric planning designed to test long-context video under- standing with 3-minute videos.
4.3. Long-Term Open-Ended Question Answering
We now investigate the performance of our models on open- ended question answering using the MovieChat-IKK dataset (Song et al., 2023), a benchmark comprising long videos with an average duration of around 8 minutes. We compare our models with other baselines based on Vicuna-7B (Chi- ang et al., 2023), including VideoChat (Li et al., 2023b), Eddo-ChatGPT (Maaz et al., 2024), MovieChat (Song et al., 2023; 2024), MovieChat+ (Song et al., 2024). Furthermore,
Tab. presents the results of our methods compared to strong parameter-free baselines. co-Video LLaMA, equipped with the LTM module (a- = 0.9) and using sticky memories, significantly outperforms both the uniform LTM variant, also for a = 0.9 and the model without LTM (a = 1), achieving a notable accuracy improvement of +6 points over the latter, similar trend is observed with co-
6

---

2o-VIDE: A Training-Free Approach to Long Video Understanding via Continuous- Time Memory Consolidation
Table MovieChat Results. Score measures the overall answer score, stands for correctness information, DO stands detail orientation, and CU stands for contextual understanding. We bold the best results and underline the best within category. We omit the temporal and consistency metrics due the absence of subset specific to these metrics. Except Moviechatt, results were taken from (Song al., 2023).
Method
LLM
Number of Frames
Accuracy
Score
C1
DO
CU
Video Chat (Li al., 2023b)
Vicuna-72 BB
32
61.0
3.34
3.26
3.20
3.38
(Maaz et al., 2024)
Vicuna-78 MB
100
44.2
2.71
2.48
2.78
3.03
Video LLaMA-Baxse Models
Video LLaMA (Zhang et al., 2023b)
Vicuna-78 -
32
51.4
3.10
3.30
2.53
3.28
MovicChat (Song ct al., 2023)
Vicuna-78 7B
2048
67.8
3.81
3.32
3.28
3.44
MovieChat+ (Song al., 2024)
Vicuna-7B
2048
66.4
3.67
3.70
3.30
3.62
Ve-Video LLaMA (no LTM)
ficuna-7B
2048
68.0
3.76
3.72
3.33
3.71
00 0o-Video LLaMA (uniform)
Vicuna-22 7B
2048
66.5
3.69
3.60
3.31
3.58
Vo-Video LLaMA (sticky)
Vicuna-7B
2048
72.2
3,88
3,89
3.47
3.79
VodVidoo LLaMA (no STM uniform)
Vicuna- 7B
2048
62.4
3.75
3.36
3.38
3.52
Vo-Video LLaMA (no STM sticky)
Vicuna-77 -B
2048
59.2
3.68
3.30
3,30
3.44
VideoChat2-Baie Based Models
VidcoChat2
Mistral-78
16
62.2
3.72
3,46
3,60
3.69
00- VideoCChhh (no L.TM)
Mistral-7B
128
63.9
3.74
3.54
3.60
3.73
VideoChat2 (uniform)
His
128
64.1
3.73
3.54
3.60
3.75
00- VideoChat2 (sticky)
Mistral-7B -B
128
63.9
3.74
3.55
3.63
3.74
oo-
VideoChat2 (no STM uniform)
Mistral-7B
128
65.7
3.78
3.65
3.60
3.84
00- VideoChat2 (no STM sticky)
Mistral-7B
128
3.85
3.71
3.68
3.96
we evaluate our xc- VIDEO variants with a 0.9 for both uniform sampling and sticky memories, well with 0 (i.e., without STM). Following the standard evalua- tion method for open-ended questions, prompt GPT-3.5 (OpenAl et al., 2024) for yes/no answer prediction, con- fidence score (0 to 5), and other qualitative metrics. The prompts are shown in App. A.3.
4.4. Qualitative Analysis
In Fig. 3, we show the continuous attention density map over the LTM in the final layer of the video Q-Former for the last chunk of co- V-DDO LLaMA. The example uses the Interstellar trailer, divided into 8 chunks of 256 frames each, with 0.75, N 1024, and a 0.9. The bottom heatmap reveals that the continuous attention favours frames after = r, where the sticky LTM exhibits peaks before this point. contrast, the uniform LTM shows vanishing density as t decreases, likely due context contraction across chunks, which might explain the superior results for sticky memories.
As shown in Tab. 3, our 0o- VIDEO LLaMA with sticky memories outperforms all models in its category, as well VideoChat and Video-ChatGPT, across all metrics. It also surpasses MovieChat, which is designed for training- free long-context video understanding. In contrast, oo- Video I.LMMA with uniform sampling performs worse than both sticky memories and the model without LTM. Ad- ditionally, using only the LTM does not improve perfor- mance, highlighting that a weighted combination of STM and LTM yields the best results for the Video-LLaMA cate- gory. The same does not apply the VideoChat2 category, where replacing the STM with the LTM yields top results. For a values other than O, the performance of co-V VIDOO with VideoChat2 remains nearly unchanged. Surprisingly, VideoChat2 underperforms compared to the Video LLaMA category despite being trained on a larger dataset. We hy- pothesize this is due to differences in the training datasets: Video LLaMA was trained video descriptions with mul- tiple sentences, encouraging more context, which increases the probability of correct predictions, while VideoChat2 was fine-tuned on concise datasets, favouring brief, open-ended predictions. Ablation studics on 0o- VI-VIDEO with Video LLaMA are presented in App. B.I.
In Fig. 4, we present the attention density as function of the number of frames, with a = 0.9, N = 256, and - 0.5, for chunks of 256 frames each, spanning 3 contraction steps. We also display 6 representative frames from high-density regions by identifying the top 26 frames within each interval and selecting the 6 non-redundant frames. The selected frames appear align with visually striking narratively significant scenes within the trailer, which suggests that oo- VIDEO effectively might capture key moments in the video and discard irrelevant parts. For example, in the final interval, the region with the lowest attention density corresponds the credits section the video. We show similar figure but for the uniform sampling in App. B.2.
5. Related Work
There are several recent advances in long-context video understanding (Li et al., 2023d; Liu et al., 2024a; Balaze- vie et al., 2024; Wang al., 2024b; Shu et al., 2024; Ye
7

---

D0- VG-VIDEO: A Training-Pree Freee Approach to Long Video Understanding via Continuous- Time Memory Consolidation
Sticky
230090
3.30
00000
2,005
00
03
1.0
Figure 3. (Top) LTM attention density on the [0, interval for the Interstellar trailer, using sticky memories in the final chunk of the 00- Vi-Vodoo LLaMA video Q- former'ss last layer. (Bottom) The same attention density map, extended over the full interval.
0.00160
0,00160
0,0008
0.00155
0.00150
0.0007
102
154
236
256
307
358
430
512
512
563
6.0
73.7
70B
Frames
Framess
Frames
12
128
311
339
400
528
545
148
150
191
583
555
571
Figure 4. Highest continuous attention density frames selected using sticky memories in the Interstellar trailer for oo- Video LLaMA
across 3 chunks. (Left) Interval: [0, (Middle) Interval: (7" 77. (Right) Interval: (T, 1].
et al., 2024; Chen et al., 2024b), but few adapting trans- formers to leverage temporal information in training-free setting, Closest to our work is Song et al. (2023; 2024), which extends Video LLaMA with memory consolidation by using a heuristic to merge similar frames, which alters frame embedding-level information. In contrast, our method retains embedding integrity, equipping the video Q-former's cross-attention with an LTM to efficiently process an arbi- trary number of frames in one single pass over the video.
6. Conclusions
We introduced lightweight extension short-video vision LLMs, enabling arbitrary-long video understanding by aug- menting the video Q-former's cross-attention mechanism with a long-term memory module that consolidates global information dynamically. Our approach adapts continuous attention to perform visual memory consolidation, allocat- ing higher granularity the most relevant frames. This ensures an efficient and focused representation of critical moments while maintaining scalability. Additionally, our method enables the sequential processing of videos with single pass. Despite being training- training-free, our approach also paves the way for scalable long-context video understanding with transformers as spatio-temporal feature extractors.
Moreover, works such as Zhang et al. (2024), Shu et al. (2024), and Chen et al. (2024b) address the challenge of long video contexts. However, these approaches involve fine-tuning training models from seratch, which can be computationally expensive and time-intensive. Our ap- proach, in contrast, enables the seamless adaptation of short video models to arbitrary long contexts without the need for sparse subsampling or discarding important information.
Our work takes inspiration from cognitive and mechanistic theories of memory (re)consolidation in brains (Hardt et al., 2010; Preston & Fichenbaum, 2013; Ma et al., 2014), and deeper integration with such theories may be pursued. For example, memory reactivation or "replay" is thought to be key component of systems consolidation in "offline" states such as sleep. Our model could be extended to incorporate such with further training and schema-driven fine-tuning for the purposes of continual learning (Cai et al., 2024). Furthermore,
Our approach builds on continuous attention mechanisms, originally proposed by Martins al. (2020), and later ap- plied image, speech, and natural language processing tasks (Farinhas al., 2021; Martins al., 2022a;b). We extend these ideas to video data by replacing the continuous softmax with the Gibbs PDF, which better replicates the discrete softmax used in the video Q-Former attention.
neural architecture, model goes be- yond current brain models of episodic memory processing, which focus on discrete low-dimensional sequences of static images and relatively simple functionalities such as mem-
8

---

0o-VIDE: A Training-Free Approach to Long Video Understanding via Continuous-Ting Memory Consolidation
ory recall and event segmentation (Franklin al., 2020; Chandra et al., 2025). Given model's integration of rich and streaming visual input with flexible and sophisticated querying via text prompting, interpretability analyses of our architecture may provide insights regarding how episodic memory may be interrogated for complex inferences in the human brain (Tulving, 2002; Radvansky Zacks, 2014).
language model for understanding, localization, text read- ing, and beyond. arXiv preprint wrXiv:2308./29000 2023.
Balazevic, I., Shi, Y., Papalampidi, P., Chaabouni, R., Kop- pula. S., and Henaff, O. Memory consolidation enables long-context video understanding. In Salakhutdinov, R., Kolter, Z., Heller, K., Weller, A.. Oliver, N., Scarlett, J., and Berkenkamp, F. (eds.), Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 2527-2544 PMLR, 21-27 Jul 2024.
Impact Statement
We discuss the broader implications of our work, including ethical considerations and potential societal consequences. Our framework extends the capabilities of existing short- context multimodal language models, enabling them to pro- cess unbounded video contexts without requiring retraining. This is particularly relevant given concerns about the energy consumption of training large models (Strubell et al., 2019). However, we must also acknowledge the societal risks as- sociated with video models, especially their potential use in privacy-violating surveillance. As our approach enables scaling to longer videos, there a concern that it could be applied in undesirable domains. While current state-of-the- art models are often trained datasets with documented biases, intentionally focus applications using standard benchmark applications, aiming to distance ourselves from harmful uses.
Brady, T. F., Konkle, T., Alvarez, G. A., and Oliva, A. Visual long-term memory has a massive storage capacity for object details. Proceedings of the National Academy of Sciences, 105(38):14325-11 14329, 2008. doi: 10.1073/ pnas.0803390105.
Brown, P. J. and Zidek, J. V. Adaptive multivariate ridge regression. The Annuals of Statistics, 1980.
Cai, C., Wang, Z., Gao, J., Liu, W., Lu, Y., Zhang, R., and Yap, K.-H. Empowering large language model for continual video question answering with collaborative prompting, 2024. arXiv:2410.00771V2.
Carr, M. F., Jadhav, S. P., and Frank, L. M. Hippocampal replay in the awake state: A potential substrate for mem- ory consolidation and retrieval. Nature Neuroscience, 14: 147-153, 2011.
Acknowledgments
We would like thank Marcos Treviso, Giuseppe Attana- sio, Sweta Agrawal, Chryssa Zerva and the SARDINE lab team for helpful discussions. This work was supported by EU's Horizon Europe Research and Innovation Actions (UT- TER, contract 101070631), by the project DECOLLAGE (ERC-2022-CoG) 101088763), by the Portuguese Recovery and Resilience Plan through project C645008882-00000055 (Center for Responsible AI), and by FCT/MECI through national funds and when applicable co-funded EU funds under UID/50008: Instituto de Telecomunicações.
Chandra, S., Sharma, S., Chaudhuri, R., et al. Episodic and associative memory from spatial scaffolds in the hippocampus. Nature, XX(XX)XX-XXX 2025. doi: 10. 00.1088/441586-024-03399
Chen, L., Wei, X., Li, J., Dong, X., Zhang, P., Zang, Y., Chen, Z., Duan, H., Lin, B., Tang, Z. al. Sharegpt4video: Improving video understand- ing and generation with better captions. arXiv preprint arXiv:2406.04325, 2024a.
Chen, Y., Xue, F. Li, D., Hu, Q., Zhu, L., Li, X., Fang, Y.. Tang. H., Yang. S., Liu, Z., He, E., Yin, H., Molchanov, P., Kautz, J., Fan, L.., Zhu, Lu, Y., and Han, S. Longvila: Scaling long-context visual language models for long videos, 2024b.
References
Atreja, S., Ashkinaze, J., Li, L., Mendelsohn, J., and Hemphill, I.. Prompt design matters for computational so- cial science tasks but in unpredictable ways, 2024. URL ht https::/ //arxiv..iiii///a wwwpxv.vvg//ee. abs / 2406. 11980.
Cheng. Z., Leng. S., Zhang, H. Xin, Y., Li, X., Chen, G., Zhu, Y., Zhang, W., Luo, Z., Zhao, D., and Bing, L. Videollama 2: Advancing spatial-temporal mod- eling and audio understanding in video-llms. arXiv preprint 240x::2406.07476, 2024. URL https:// arxiv. org/ abs 240.
Bahdanau, D., Cho, K., and Bengio, Y. Neural machine translation by jointly learning to align and translate. In Proc. of International Conference on Learning Represen- lutions, 2015.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J. E., and et al. Vicuna: An open- chatbot impressing gpt-4
Bai, J.. Bai, S., Yang, S.,. Wang. S., Tan, S., Wang. P., Lin, J., Zhou, C., and Zhou, J. Qwen-vl: versatile vision-

---

00- V--VIDE A Training-Free Approach to Long Video Understanding via Continuous- Memory Consolidation
with 90% chatgpt quality. https://vicuns: Imsys.
G., Lample, G., Saulnier, L.,, Lavaud, L. R., Lachaux, M.-
org, 2023.
A., Stock, P., Scao, T. L.,, Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023.
Cowan, E. T., Schapiro, A. C., Dunsmoor, J. E., and Murty, V. P. Memory consolidation as an adaptive pro- cess. Psychonomic Bulletin & Review, 28:1796-1810, 2021. doi: 10.3755K113223-00--119993 URL https: //doi. org/ 3eepe//.....////81 /1////113333333 01971-0
Jin, P., Takanobu, R., Zhang, C., Cao, X., and Yuan, Chat-umivi: Unified visual representation empowers large language models with image and video understanding. arXiv preprint arXiv:231 2023.
Dosovitskiy, A., Beyer, I.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Li, J., Li, D., Savarese, S., and Hoi, S. BLIP-2: Boot- strapping inguaage-image pre-training with frozen image encoders and large language models. In Proceedings of the 40th International Conference on Machine Learning. 2023a.
Li, K.,, He, Y., Wang, Y., Li, Y., Wang, W., Luo, P., Wang, Y., Wang, L.,, and Qiao, Y. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305. 06355, 2023b.
Dudai, Y., Karni, A., and Born, J. The consolidation and transformation of memory. Neuron, 88:20-32, 0200022222000 2015.
Fang. Y., Wang, W., Xie, B., Sun, Q., Wu, L., Wang, X., Huang, T., Wang, X., and Cao, Eva: Exploring the limits of masked visual representation learning at scale. 2022.
Li, K., Wang, Y., Li, Y., Wang, Y. He, Y., Wang, L., and Qiao, Y. Unmasked teacher: Towards training- efficient video foundation models. 2023 IEEE/CVF In- ternational Conference on Computer Vision (ICCV), pp. 19891-19903, 2023c.
Farinhas, A., Martins, A. F. T., and Aguiar, P. M. Q. Multi- modal continuous visual attention mechanisms. Pro- ceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pp. 1047-1056, Oc- tober 2021.
Li, K. Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., Wang, L., and Qiao, Y. Mvbench: A comprehensive multi-modal video under- standing benchmark, 2024. URL https: attps://arxiv. //rxxv...vvvvv org/ abs/ 2311. 17005.
Frankland, P. W. and Bontempi, B. The organization of re- cent and remote memories. Nature Reviews Neuroscience, 6(2):119-130) 2005.
Li, Y., Wang, C., and Jia, J. Llama-vid: An image is worth 2 tokens in large language models, 2023d.
Franklin, N. T., Norman, K. A., Ranganath, C., 7acks, J. M., and Gershman, S. J. Structured event memory: A neuro- symbolic model of event cognition. Psychological Re- view, 1 127(3):327-361) 2020. doi: 10. 00.10377ex0000177.
Liu, H. Li, C., Wu, Q., and Lee, Y. Visual instruction tuning, 2023.
Liu, J., Wang. Y., Ma, H., Wu, X., Ma, X., Wei, X., Jiao, J., Wu, F., and Hu, J. Kangaroo: A powerful video-language model supporting long-context video input, 2024a.
Fu, C., Dai, Y., Luo, Y., Li, L.,, Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., Chen, P., Li, Y., Lin, S., Zhao, S., Li, K., Xu, T., Zheng, X., Chen, E., Ji, R., and Sun, X. Video-mme: The first-ever comprehensive evalu- ation benchmark of multi-modal Ilms in video analysis, 2024.
Liu, R., Li, C., Tang, H., Ge, Y., Shan, Y., and Li, G. St-lim: Large language models are effective temporal learners, 2024b.
Liu, Z.. Zhu, L.. Shi, B., Zhang, Z., Lou, Y., Yang, S., Xi, H., Cao. S., Gu, Y., Li, D., Li, X., Fang, Y., Chen, Y.. Hsieh, C.-Y,, Huang, D.-A., Cheng, A.-C.,, Nath, V., Hu, J., Liu, S., Krishna, R., Xu, D., Wang, X., Molchanov, P., Kautz, J., Yin, H., Han, S., and Lu, Y. Nvila: Efficient frontier visual language models, 2024c.
Hardt, O., Einarsson, E. O., and Nader, K. A bridge over troubled water: reconsolidation as a link between cog- nitive and neuroscientific memory research traditions. Annual Review of Psychology, 61:141-167, 2010. doi:
10.1 14144400eeeeyyyyy.00000000.... 100455.
hwchase17. Langchain, 2023. URL pettps:/gitthng Accessed: 2023-12-
Luo, R., Zhao, Z. Yang, M., Dong, J., Qiu, M., Lu, P., Wang, T., and Wei, Z. Valley: Video assistant with large language model enhanced ability, 2023.
com/ Sonm///wchase17/angghhe 20.
Jiang, A. Q.. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel,
Ma, W., Husain, M., and Bays, P. Changing concepts of working memory. Nature Neuroscience, 17:347-356.
10

---

VI-VIDO A Training-Free Approach to Long Video Understanding via Continuous- Memory Consolidation
2014. doi: 10.1038/mn.365. org/10.100 1.0.0..000 36555
URL https://dol:
Deville, D., Dhar, A., Dohan, D., Dowling, S., Dunning.
S., Ecoffet, A., Eleti, A., Eloundou, T., Farhi, D., Fedus, L., Felix, N., Fishman, S. P.,, Forte, J., Fulffrdd I., Gao, L., Georges, E., Gibson, C., Goel, V., Gogineni, T., Goh, G., Gontijo-Loppes, R., Gordon, J., Grafstein, M., Gray, S., Greene, R., Gross, J., Gu, S. S., Guo, Y., Hallacy, C., Han, J., Harris, J., He, Y., Heaton, M., Heidecke, J., Hesse, C., Hickey, A.., Hickey, W., Hoeschele, P., Houghton. B.. Hsu, K., Hu, S., Hu, X., Huizinga, J., Jain, S., Jain, S., Jang, J., Jiang, A., Jiang, R., Jin, H., Jin, D., Jomoto, S.. Jonn, B., Jun, H., Kaftan, T., Eukasz Kaiser, Kamali, A., Kanitscheider, I., Keskar, N. S., Khan, T., Kilpatrick, L., Kim, J. W., Kim, C., Kim, Y., Kirchner, J. H., Kiros, J., Knight, M., Kokotajlo, D., Lukasz Kondraciuk, Kondrich, A., Konstantinidis, A., Kosic, K., Krueger, G., Kuo, V., Lampe, M., Lan, I., Lee, T., Leike, J., Leung, J., Levy, D.. Li, C., Lim, R., Lin, M., Lin, S., Litwin, M., Lopez, T., Lowe, R., Lue, P., Makanju, A., Malfacini, K., Manning, S., Markov, T., Markovski, Y., Martin, B., Mayer, K., Mayne, A., McGrew, B., McKinney, S. M., McLeavey, C., McMillan, P., McNeil, J., Medina, D., Mchta, A., Menick, J., Metz, L.., Mishchenko, A., Mishkin, P., Monaco, V., Morikawa, E., Mossing, D., Mu, T., Murati, M., Murk, O., Mély, D., Nair, A., Nakano, R., Nayak, R., Neelakantan, A., Ngo, R., Noh, H., Ouyang, L.., O' Keefe, C., Pachocki,
Maaz, M., Rasheed, H., Khan, S., and Khan, F. S. Video- chatgpt: Towards detailed video understanding via large vision and language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), 2024.
Mangalam, K., Akshulakov, R., and Malik, J. Egoschema: A diagnostic benchmark for very long-form video language understanding. arXiv preprini arXiv:2308.00 09166, 2023.
Martins, A., Farinhas, A., Treviso, M., Niculae, V., Aguiar, P., and Figueiredo, M. Sparse and continuous attention mechanisms. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 20989-21001. Curran Associates, Inc., 2020. URL htttp:://prcceeednggg.o neurips. cc
f0b76267 0117//22770222213333222222233333 Pape. pdf.
Martins, A. F. T., Treviso, M., Farinhas, A., Aguiar, P. M. Q., Figueiredo, M. A. T., Blondel, M., and Nicu- lae, V. Sparse continuous distributions and fenchel-young losses. Journal of Machine Learning Research, 23(257); 1-74, 2022a. URL http: /ttp:://mmrr../ http:///inno.rr/ppeeeeeeee org/papers/ v23 005/21-0877 html.
J., Paino, A., Palermo, J., Pantuliano, A., Parascandolo, G., Parish, J., Parparita, E., Passos, A., Pavlov, M., Peng, A., Perelman, A., de Avila Belbute Peres, F., Petrov, M., de Oliveira Pinto, H. P., Michael, Pokorny, Pokrass, M., Pong. V. H. Powell, T., Power, A., Power, B., Prochl, E., Puri, R., Radford, A., Rae, J., Ramesh, A., Raymond, C., Real, F., Rimbach, K., Ross, C., Rotsted, B., Roussez, H., Ryder, N., Saltarelli, M., Sanders, T., Santurkar, S., Sastry, G., Schmidt, H., Schnurr, D., Schulman, J., Sel- sam, D., Sheppard, K., Sherbakov, T., Shieh, J., Shoker, S., Shyam, P., Sidor, S., Sigler, E., Simens, M., Sitkin, J., Slama, K., Sohl, I., Sokolowsky, B., Song. Y., Stau- dacher, N., Such, F. P., Summers, N., Sutskever, I., Tang, J., Tezak, N., Thompson, M. B., Tillet, P., Tootoonchian, A., Tseng, F., Tuggle, P., Turley, N., Tworek, J., Uribe, J. F. C., Vallone, A., Vijayvergiya, A., Voss, C., Wainwright, C., Waang J. J., Wang. A., Wang, B., Ward, J., Wei, J., Weinmann, C., Welihinda, A., Welinder, P., Weng, J., Weng, L., Wiethoff, M., Willner, D., Winter, C., Wolrich, S. Wong, H., Workman, L., Wu, S., Wu, J., Wu, M., Xiao, K., Xu, T., Yoo, S., Yu, K., Yuan, Q., Zaremba, W., Zellers, R., Zhang, C., Zhang, M., Zhao S., Zheng, T., Zhuang, J., Zhuk, W., and Zoph, B. Gpt-4 technical report, 2024.
Martins, P. H., Marinho, Z., and Martins, A. F. oo-former: Infinite memory transformer. In Proc. ACL, 2022b.
McGaugh, J. Making lasting memories: Remembering the significant. Proceedings of the National Academy of Sci- ences of the United States of America, 110110402-10477,
2013. doi: 10.1073/pnas. 13012091 10. URL https: //doi. . orooo/gg///00 /10 1 073 pnas. , 13012091 209110.
McNamee, D. C. The generative neural microdynamics of cognitive processing. Current Opinion in Neurobiol- ogy, 85: 102855, 2024. doi: (0.1016/j.comb.200 wew.b.024.10285555 102855. Epub 2024 Feb 29.
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., Avila, R., Babuschkin, I., Bal- aji. S., Balcom, V. Baltescu, P., Bao, H., Bavarian, M., Belgum. J.. Bello, I., Berdine, J., Bernadett Saauddeett-hairo.... G., Berner, C., Bogdonoff, L., Boiko, O., Boyd, M., Brakman, A.-L,, Brockman, G., Brooks, T., Brundage, M., Button, K., Cai, T., Campbell, R., Cann, A., Carey, B., Carlson, C. Carmichael, R., Chan, B., Chang, C., Chantzis, F., Chen, D., Chen, S., Chen, R., Chen, J., Chen, M., Chess, B. Cho, C., Chu, C., Chung, H. W., Cummings, D., Cur- rier, J., Dai, Y., Decareaux, C., Degry, T., Deutsch, N.,
Preston, A. R. and Eichenbaum, H. The interplay of hip- pocampus and prefrontal cortex in memory-based deci- sion making, Current Biology, 23(11):R744-R77 2013.
Radvansky, G. A. and Zacks, J. M. Event Cognition. Oxford

---

0o-VIDE: A Training-Free Approach to Long Video Understanding via Continuous- Time Memory Consolidation
University Press, New York, NY, 2014.
computer vision and paltern recognition, pp. 9777-9786. 2021.
Shu, Y. Liu, Z., Zhang, P., Qin, M., Zhou, J., Liang. Z., Huang, T., and Zhao, B. Video-xl: Extra-long vision lan- guage model for hour-scale video understanding, 2024.
Ye, J., Xu, Li, Hu, A., Yan, M., Qian, Q. Zhang, J., Huang, F., and Zhou, J. mplug-ow13: Towards long image-sequence understanding in multi-modal large lan- guage models, 2024.
Song, E.,, Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Guo, X., Ye, T., Lu, Y., Hwang, J.-N., et al. Moviechat: From dense token sparse memory for long video un- derstanding. arXiv preprint arXiv:2307. (6449, 2023.
Zhang, C., Lu, T., Islam, M. M., Wang. Z., Yu, S., Bansal, M., and Bertasius, G. A simple Ilm framework for long- range video question-answering 2023a.
Song. E., Chai, W., Ye, T., Hwang, J.-N., Li, X., and Wang, G. Moviechat+: Question-aware sparse mem- ory for long video question answering. arXiv preprint arXiv:2404.17176. 2024.
Zhang, H., Li, X., and Bing, L. Videollama: An instruction- tuned audio-visual language model for video understand- ing. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP 2023), 2023b.
Strubell, E., Ganesh, A., and McCallum, A. Energy and policy considerations for deep learning in NLP, In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 3645-3650, Florence, Italy, July 2019. Association for Computa- tional Linguistics. doi: 10.18653/v1//19-11555 URL https:/ / aawaaaanntlllggooooee hwtp:19-g//////
Zhang, P., Zhang, K., Li, B., Zeng, G., Yang, J., Zhang, Y., Wang. Z., Tan, H., Li, C., and Liu, Z. Long con- text transfer from language vision. arXiv preprint arXiv:2406.166522 2024. URL https://arxiv: org
Tulving, E. Episodic memory: From mind to brain. Annual Review of Psychology, 53:1-25, 2002. doi:
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint wrXhv23344.0552 2023.
10.1 1446/a/mueeyyyccch...0000000000
URL pettps:/dooi.oo wttp:////....1///areeeeeeeeeeeee psych. 53.100901.135111
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, L. u., and Polosukhin, Attention is you need. Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R. (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: /tttps::/prcceed:/i meerii neurips. cc cc/paper /feaaerrfflee//sepe////////// 3ffe2233B4cce111la3cccccllaaaaa
pdf.
Wang, X., Zhang, Y., Zohar, O., and Yeung-Levy, S. Videoa- gent: Long-form video understanding with large language model as agent. European Conference on Computer Vi- sion (ECCV), 2024a.
Wang, Y., Xie, C., Liu, Y., and Zheng, Z. Videollamb: Long video understanding with recurrent memory bridges. arxiv, 2024b.
Wang. Z., Yu, S., Stengel-Eskin, E., Yoon, J., Cheng, F,, Bertasius, G., and Bansal, M. Videotree: Adaptive tree- based video representation for IIm reasoning on long videos. arXiv preprint arXiv:2405. 19209, 2024c.
Xiao, J., Shang, X., Yao. A., and Chua, T.-. Next-qa: Next phase of question-anseee answering to explaining temporal actions. In Proceedings of the IEEF/VVF conference on
12

---

0o-VIDEO: A Training-Free Approach to Long Video Understanding via Continuous- Time Memory Consolidation
A. Implementation Details and Hyperparameters
A.1. Additional Implementation Details
Video LLaMA-Based Models. As discussed in $3, Vidco-LLaMA (Zhang et al., 2023b) serves as one of the adapted models in this work. employs a cascade of two Q-formers one for spatial feature extraction and the other for temporal feature extraction-with the latter enhanced by our LTM integration. The video Q-former and projection layer parameters consistent with the Lateeo-LLaMM--------ieeeee tuning data from MiniGPT-4 (Zhu et al., 2023), LLaVA (Liu et al., 2023), and VideoChat (Li et al., 2023b). For visual feature extraction, we utilize the ViT-G/14 encoder from EVA-CLIP (Fang et al., 2022), while spatial dependency features rely on the Q-former from BIJP-2 (Li et al., 2023a). This lightweight ViT facilitates efficient processing of a large number of frames per chunk. The LLM used this model is Vicuna-78 (Chiang et al., 2023).
model (Zhang al., 2023b), which was fine-tuned using instruction-
Atreja al. (2024) and our empirical validation have shown limitations in respecting the format of the multiple-choie answer asked by the prompt, influencing the accuracy this method in multiple-choice datasets. To address this, and for these datasets follow the approach of (Song et al., 2023; 2024) and provide modified model, co- So-Video LLaMA, exclusively with the questions in the prompt. Using LangChain (hwchasel7, 2023), we calculate the similarity between oo- VidVideo LLaMA's open-ended responses and the given options, selecting the option that best aligns with the expected answer.
Building Video-LLaMA, also evaluated our methods using VideoChat2 (Li et al., 2024), more advanced short-video model. Like Video-LLaMA, it incorporates video Q-former, but it benefits from additional instruction-tuning data (Li et al., 2024). For visual encoding, we use UMT-L (Li et al., 2023c), which captures both spatial and temporal features specifically designed for video data. This model's higher memory requirements necessitated the use of smaller frame chunks in our experiments. For the LLM component, we used the stage-3 3 Mistral-7B version of VideoChat2, 4 which demonstrated the best performance in its original paper.
VideoChat2-BBl Models.
A.2. Hyperparameters
We report in Tab. 4 the hyperparameters used in our experiments. For co- Video LLaMA and for NeXT-QA we use all the frames availables with variable number of chunks of 256 frames.
Table 4. Hyperparameters used in vorvaniants for the different datasets.
Parameter
NeXT-QA
Egoschema
VideoMMEE cc- Sc-VideoChat2
MovieChat
oc- Video LLaMA
co Sc-VideoChut2
oc- Video LLaMA
S--iideChhat2
oc- Video LLaMA
cc-VideuChat2
# chunks
8
8
8
8
8
8
# frames
256
256
256
256
256 0.75
1024
256
256
1024
256
0.75
0.75
0.75
0.5
0.75
0.75
A.3. Evaluation
We further show in List. 1, 2, 3, 4 the prompts used for evaluation on open- open-ended question answering tasks for the Moviechat dataset.
Listing ChatGPT-3.5 prompt for overall accuracy and score metric.
"role": "systen",
"conten:":
*You are an intelligens chatbot designed for avalusting the cocrsstnecs ot gencrative cutputs for quastion-anever peice.
*Your task in to conaaee the predicted answer with the cccrect answer and determine it they match meaningtully. Here's how you can accomplish the taak:"
*#F:NSTRUCTIONS:
Pecus on the meaningtul natch between the predicted answer and the correct answer.An"
Concidor eynоаyms or passphrases as valid matches. \a*
Evaluate the cocrectress or the prediction compared the ansver."
13

---

V-VVDDO A Training-Free Approach to Long Video Understanding via Continuous- Time Memory Consolidation
"role": "user",
"conscac":
*Flease evaluate the tolloring video-based qusstion-answer pairrlain"
("quaation: [quaation]1et
("Corrost Answez: (ansver/
""Predicted Answer: Anknn
integor value betwoon and with indicatiae sha "Flease generate the response in the tor ot a Python dictionaty string with keys "pres! and "scoze", whore value at 'pred" is a string of 'yes' oz "no" and value of "acoze" is in INIBGER, not SCRING." "Do NOT PROVIDE MY OTHER CUTPUT TEXT EXPLAMArION. Only provide tao Python dieticnery string.
*Frovida your evaluation only yes/no and where highest necningtol natch.
*For example, your reaponse should look like this: I'predt: : "yesf, facorafa 4.B)."
Listing ChatGPT-335 prompt for the contextual understanding (CT) metric.
"roln": "syster",
"conienn "
*You, an intelligens chatbot designed for evalusting the factual accuracy of generative outputs for viceo-based guestion-anewer pains,
*Youz Lask Lo compare Lhe predicted ansver with the correat anaver and deteoning if thay are factually conxlatent. Eazefa hox you can accomplish the tasks"
*#ENSSTRCT11NS::
Poous on the Faccual const stenoy betweer the pted answer and the correct anewer, The predicted snewer should not contain any mi misintergg interrretat lons or minonforee inn
*-
The predictes answer
ractually sccurate align with the viceo content.in"
che. Rvaluate the faccual accurany of the predict on oungared to the answes.
Conaider synonyns or paraphra sea
1.
"rolo": "uccr",
"conten:":
*Please evallaat the fol loxing viden-bazed quest fon-awwer peirthain"
r"goestion: Iquestion/
#CCorrect Anewer: lansveriAn"
P*Predi cted knswers Igred/Ankn*
*Frovide your evaluation only ds fectual accuracy score where the ractual accuracy score is an intoger value zetweea 0 and 5, highest level tactusl corsistency. of Pyshon di ioneryy string wich keys 'acore', where its value Factual acmaray scoze in INTEGER, not STRING.*
with indicating
Plleas generate the response
*0 NOT PROVIDE MYY OTHER OUTPUT TEXT OR EXPLANATION, only provide the Python dictionery string,
*For example, your respease should look like this: I sesrer 0.8)."
Listing 3. CharGPT-3.5 prompt for the detailed orientation (DO) metric.
"rolw"; "syster", "conteni";
'You are an intelligen; chatbat desicned for avaluating the detail ortentacion of generative outputs foz video-based quastion-
answer pairs.
*Youz task is to conpare the predicted answer with the correct anaver and deternine ita level of detail, considering both completonass and apecificity. Harets you accomplish the taski"
*4CXSTROCTIONS:
Check If the predicsed answer novers all najor pol ntw from the video. The responee should not Teave out any key axpecte, Art includes specific detaila zather chan just generio puints. should provide composhenn information ion that is tied to apecific elenenta the video.
Roal uele wheller the predicted
Donsider syrosyss or pa paraphrases as valid not An*
Peavide aingle evalustion souce that awflecta the level of detail inn of Uhe predict lun, cnn comsidering buth completeness and specifisity."
1.
"role": "oser", "gont
*lleas avaluate Lhe fullowing video based queat iun anaver paieskain"
inguestion: iqrestior Aee
reCorrect Answer: [araver] An*
d"Predieted Answess [pred]
*Provide your eval uatiotoo for only as a d orientation soore where the detail orientation acore is an integer value betweer: and S, with 5 indicst ing the highest level of detail orientacion
where its value detail *0 Nor PROVInE ANY OTHER ourPOr "EX" RXPLANAtION. only provide the Python diet Sintlonwry string, *For example, your responce should loss 1ike thiss sorsereefs 6.01."
*Flease generate the response in the
Python dictionary string with keys 'score', y inntationn soore in INTRSER, nos STRING."
Listing 4. CharGPT-3.5 prompt for the contextual understanding (CU) metric.
"role't: "ayaten",
"ccazeac":
*You are intelligen: chatbot designed for avalusting the contextual urderstanding of generativs outpuce for vidco-azsed quastion -answez pairs.
*Your task is to conaare the predicted answer with the correct answer and dstermino if the goncrated respease aligas with the
14

---

X-VIDEO A Training-Free Approach to Long Video Understanding via Continuous- Time Memory Consolidation
overall context the video concent, Here''s now you can accomplish the toek:"
*
Evaluate whother prediaced answez allens
conzext video content. not provido
that is out of context or misaligned, in"
The ctedicted anover must septure the main thames and sentimente of she ind
Censider syseayna
catches.Aat
Provido your ovalcation the contextuaa undarstanding the prodiction compared ansvar."
"role": "asce",
*consen:":
*Flcase evaluate the tolloxing video-based quastior-anower pairslatn"
ingocstion: (quastior] le"
MCCorrett Answer: [anaver]
("Predicted Answez: (alad
"Ecorid: your evaluation only 25 contexcual understanding seare where the contextual undorstanding score in an integer valce backeen and 5, with 5 indinating the hishest level of contextual underatandiae,
'scoze', value contoxtucl too Nor PROVIDE ANY OTHER CUTPOT TEXP OR EXPLANAIONN, only provide the Python dictionary atring, *For exampie, your responce should look like this: ("cserre's Q.8)."
"Flaaso
Pyshon dicticnary string STRING.
underalanding scuse INTEGRR,
B. Additional Experiments
B.1. Ablation Studies on oo- VodVideo LLaMA
0.72
Uniform Sticky
3.85
0.70
3.80
0.68
3.75
3.70
Accuracy
512
1024
Scare
N
0.725
0.700
3.85
0.675
3.80
0.650
3.75
0.625
0.600
3.70
0.25
0.9
0.25
0.75
Figure Ablation studies on MovieChat dataset: Evaluation accuracy and score metries various values of the number basis functions N and the contribution of long-term memory
In this section, we conduct ablation studies Vo-Video LLaMA on the MovieChat dataset. We ablate three hyperparameters: the percentage of the long-term memory used a. the number of basis functions N and the sampling method. We explore E (0, 0.25, 0.5, 0.75, 0.95, 1). covering the full spectrum from exclusively using the LTM (a = 0) to exclusively using the STM (a = 1). Additionally, we vary the number of basis functions with N E {128, 256, 512, 1024] to evaluate the impact this parameter on performance and vary the sampling as either uniform or sticky.
Fig. 5 presents the accuracy and score provided by ChatGPT 3.5 as functions of the explored hyperparameters. The results reveal general trend of increasing accuracy and score with a, up to certain point, after which slight decline is observed as approaches 1. This trend is somewhat explained by the inherent variability in CharGPT's outputs. Additionally, both accuracy and score tend to be higher for sticky memories compared to uniform sampling from a = 0.75 onward, whereas uniform sampling demonstrates superior performance for lower values a. Another observed trend is that as the number of basis functions increases, both metrics improve. However, for uniform sampling, performance slightly decreases beyond
N 512.
15

---

DC- VO-VIDE:: A Training-Free Approach to Long Video Understanding via Continuous- Time Memory Consolidation
0.0018
0.00125
0.00100
00055
00075
102
205
258
256
410
512
563
717
#I Frames
Frames
Frames
16
79
120
435
444
450
602
6144
617
201
Figure 6. Highest continuous attention density frames selected using uniform memories in the Interstellar trailer for co- So-Video LLaMA
across 3 chunks. (Left) Interval: [0, (Middle) Interval: (7"; .7). (Right) Interval: (T, 1].
B.2. Qualitative Analysis
In Fig. 6, we illustrate the attention density function of the number of frames for the uniform sampling LTM configuration, using 0.9, N = 256, and = 0.5. This analysis spans 3 chunks of 256 frames each, corresponding to 3 contraction steps. Additionally, we showcase representative frames from high-density regions by identifying the top 10 frames in each interval and selecting non-redundant examples.
The results reveal that, while the model effectively identifies key moments during the first two contraction steps, disproportionately focuses on the credits scene in the final step. This behaviour contrasts with the results in Fig. where such focus is avoided.
We leave in Fig. additional examples of predictions of our modified Vo-eoeeo LLaMA. We divide the video into 8 chunks of 256 frames with N=1024, r = 0.75 and a = 0.9 both for uniform sampling and sticky memories.
16

---

D0- VI-VIDE:: A Training-Free Approach to Long Video Understanding via Continuous- Time Memory Consolidation
antoractior bualarr
min Video
Butomiin Rapiticdy
Modla
BOHEMIAN THAPSDER
User What the color jacket the marn character the vodeo?
User: What doee the matn charache''s & hair look like in the find arene?
Stidk Assistant: In the video, the san uowe lig the brasse fontei isstandinng near black bankgnouudd with a alg. The oxbe of packet in trans.
Stidk Assistant in the first soree, itr reats disnarter Ars carly twase Autr. Mr weering like atcanier ana Hore a brvis cruck the hadggoordd
Unifeem Assistant The von the video antrrig a Dvooer solee
Unitorm Assistant: first
with fair
juckt
and
the seal. He taptng the table, and Us also mindes the wall Additionnlls a men som woatingg white Joduct Stiing lible the samee will black and white thotograph th reall
table
with
Which vobtcke staible the vodao?
Sticky Assistant:
Thor e mark can reveoryyele and virgere rocile the sakee.
Usen finst performance Indoces or ontdooos?
Uniidceas Assistant: the finst mane of the alla ao sanaee cluck alg and Black bachamann with people in the tackground. I.. finally the nith some, we con ae dark and black bodkground with our dnving donen the street on might.
Slicky Assistant: Tee sike shass pareorsee dkkng place insid a dosk news So. 3 indiers
Assistanii on glaen video descopttion ditticol deternine
Where is the up pppcicip
totonaato entering?
performance valan secural light rodld pocuode tnsight.
not specitic descrigttons light
Slicky Assistants in the videa the decaioostais onter the spicodlp and the apontipp is an slach blck backguandd The smmeunts mouring grasorth, and quanokiy carryng then rets ztrasphess lens
What band's gameo of music duown video?
Uniform Assistant: the scmm of the video, ua can soe dark dky with brigtt abseaatin the canter The alaes appean is be alte and wa small arplane Rping the 1...
Sticky Audistanh rideo slevs rowd playing owsei is dark nove Sx die bund's grirs of muesic on assuesal allemuticss music
covetions be intered cron the gift's exp ressiions when feaves home?
sias, the atinn a with durrgeen inuttng
Uniform Assistant: Bast the vsins, appoas that the prissril plays tusks although exteppiclyy mentured.
regrestee consiivation dlsypsinteen risr
the She for felier and Huat foening.
how
bevng alous
Which insorumnn muin charactes play in the concest?
Uniforn Assietant: The gin in shessice look and wiree ber fethor femon and ler exgomerree Aiffesst rodl Hiscces tee coneppt of Seeily and the bova belsaen parent and ckibf de in mea foom fie oustinal the aisaliin Additionalls the salls spp pepaaance and posture spsciliod. but vadeo does depect young pal a tank top an black haziggnandd.
Sticky Assistant: the sevec the ware chenader playa the Aepiiennt
Uniforn Assistant: in the lae Iov coneet the main chamcter playinga
Figure 7. Examples of Video LLaMA answers with uniform sampling and sticky memories for short and ultra-long videos. Italicized corresponds to the correct answer while underlined corresponds to the wrong answer or hallucination.
17


Reviews:
[{"review_id": "aGI6vagD2j", "reviewer": "Reviewer_iEVz", "summary": "This paper proposes a method called “\\infty-VIDEO” to enable large multimodal language models (LLMs), originally designed for short video contexts to process arbitrarily long videos. The approach builds on top of existing “video Q-former” architectures by equipping them with a new continuous-time long-term memory (LTM) mechanism. Specifically, the authors use a “continuous attention” strategy that compresses and consolidates past chunks into a fixed dimension memory representation. Experiments on multiple video QA benchmarks, some of which include minutes-long and even hour-long videos, indicate the method can improve long-video comprehension in a training-free manner.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and suggestions. We are glad that you found our method novel, our evaluation sound, and the motivation of our work strong and intuitive. We address your main concerns below. \n\n> “The paper misses analysis about memory usage and runtime overhead (...) the scaling behavior and trade-offs could be more explicitly benchmarked.”\n\nThis is a good suggestion. We agree that it is important to analyze the memory usage and runtime overhead of our method for large videos. Our experiments focus on a fixed-dimensional memory representation that scales independently of video length. To address this concern, we have conducted additional experiments that show the trade-offs in inference time and memory usage as the number of video chunks and basis functions increases, including the impact of the LTM. We will add these results to the appendix.\n\n### Inference Time vs L\n| **L**|**Video LLaMA (s)**|**Ours, No LTM (s)**|**Ours, LTM (s)**|\n|-|-|-|-|\n|**256**|31.25|31.25|33.10|\n|**512**|36.68|35.55|37.42|\n|**1024**|38.72|40.03|41.41|\n|**2048**|OOM|51.30|55.79|\n\n### Memory Usage vs L\n|**L**|**Video LLaMA (Gb)**|**Ours, No LTM (Gb)**|**Ours, LTM (Gb)**|\n|-|-|-|-|\n|**256**|18.99|18.99|18.99|\n|**512**|22.67|18.99|19.17|\n|**1024**|30.04|19.46|19.47|\n|**2048**| OOM| 20.05|20.96|\n\n### Inference Time vs N\n|**L**|**Video LLaMA (s)**|**N**|**Ours (s)**|\n|-|-|-|-|\n|**64**|32.27|**64**|42.94|\n|**128**|33.49|**128**|40.51|\n|**256**|31.25|**256**|41.41|\n|**512**|36.88|**512**|41.68|\n|**1024**|38.72|**1024**|42.18|\n\n### Memory Usage vs N\n|**L**|**Video LLaMA (Gb)**|**N** |**Ours (Gb)**|\n|-|-|-|-|\n|**64**|16.23|**64**|19.47|\n|**128**|17.15|**128**| 19.47|\n|**256**|18.99|**256**|19.47|\n|**512**|22.67|**512**|19.48|\n|**1024**|30.04|**1024**|20.64|\n\nIn the first two tables, for $\\infty$-Video LLaMA, we use 256 frames per chunk and set $N=256$, while for the baseline (Video LLaMA), we use the total number of frames without chunking. In the last 2 tables for “Ours” we use 4 chunks of 256 frames. We use the Bohemian Rhapsody movie. OOM denotes an Out-of-Memory error on an A6000 GPU. Results show that our method maintains a constant memory footprint regardless of the number of frames, with only a small increase in inference time. We also observe that the increase in memory usage with the number of basis functions is smaller compared to the increase with the number of frames in the baseline. We hope this alleviates your concern.\n\n> “The conclusion from Table 4 is not consistent. It is better to test other Q-former video models and give a deeper analysis (...) compare with recent VLM models such as [*1, *2, *3].”\n\nPlease note that the main goal of our paper is to validate the benefits of integrating a LTM with continuous attention in a fair, training-free setting, for which the Q-former video models are a suitable choice. We focused on an apples-to-apples comparison to isolate the impact of our approach, paving the way for future integration into stronger VLMs, which keep evolving very rapidly. For more details, see our response to Reviewer SJwR (second question).\n\n> “Would segmenting the video by scene change (rather than fixed-length chunks) further boost performance?”\n\nIn our early experiments, we tried scene segmentation, but for fast-moving videos, this created small chunks that did not improve performance. Testing different scene-dividing granularities resulted in some chunks containing multiple scenes, which yielded worse results than fixed-length chunks. In any case, we find fixed-length chunks more interesting, since the ability of our method to generate sticky memories without the need for any scene segmentation suggests it is able to identify the most relevant information in the different scenes, bypassing this additional step.\n\n> “How are the hyperparameters chosen? For example, is the method sensitive to memory contraction factor? (...) How sensitive are results to the choice of chunk size vs. number of chunks? Is there an optimal chunk length for typical tasks?”\n\nThe number of frames per chunk was determined by balancing GPU memory constraints and alignment with the training setup of the base models. For VideoChat2-based models, we selected 16 frames per chunk, as this both fits within memory limits and also matches the number of frames used during the training of VideoChat2. For Video LLaMA, although the model was originally trained with 32 frames, [1] has demonstrated that increasing this to 256 frames does not degrade performance. We therefore used 8 chunks of 256 frames, which aligns with the total number of frames in MovieChat [1]. Finally, the $\\tau$ parameter was chosen to be the same as in the  $\\infty$-former paper [2].\n\n[1] Moviechat+: Question-aware sparse memory for long video question answering (Song et al., 2024)\n\n[2] $\\infty$-former: infinite memory transformer (Martins et al., 2022)", "claims_and_evidence": "- Claim: the proposed continuous-time attention can dynamically allocate higher granularity to crucial frames and effectively compress less relevant parts.\n\n  Evidence: qualitative heatmaps of the attention densities show peaks at visually distinctive or narrative-significant frames. Quantitative results indicate improvements over uniform sampling baselines.\n- Claim: sticky memories (adaptive sampling guided by prior attention) outperform simple uniform sampling.\n\n  Evidence: in multiple-choice QA tasks, sticky memories yield consistently better accuracy. In open-ended QA (MovieChat-1K), sticky memories often outperform pure uniform sampling.", "methods_and_evaluation_criteria": "Yes. The evaluation is sound, including:\n- Multiple-Choice Question Answering\n- Long-Term Open-Ended Question Answering", "theoretical_claims": "No theoretical claims are made.", "experimental_designs_or_analyses": "Yes.  The authors run experiments on NeXT-QA, EgoSchema, Video-MME and MovieChat-1K. Key analyses include accuracy gains from ablating the “sticky” memory vs. “uniform” memory. The paper also includes qualitative visualizations.", "relation_to_broader_scientific_literature": "I think the key contribution is the exploration of a training-free approach for long video understanding.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "**Strengths**\n- The paper proposes a training-free method which is novel and barely explored in this field. \n- Sticky memory sampling effectively focuses on critical frames, as demonstrated by meaningful attention heatmaps.\n- The motivation of the paper is strong and intuitive.\n\n**Weaknesses**\n- The paper misses analysis about memory usage and runtime overhead for very large videos. While the paper highlights the advantage of not storing all frames in memory, the scaling behavior and trade-offs could be more explicitly benchmarked. \n- The conclusion from Table 4 is not consistent. It is better to test other Q-former video models and give a deeper analysis.", "other_comments_or_suggestions": "- The authors are recommended to also compare with recent VLM models such as [*1, *2, *3].\n- Even the paper focuses on training-free setting, it is also recommended to compare with some trained based long-video understanding paper, such as longvila and Video-xl.\n- A breakdown of computational costs would be helpful—for example, how the inference time scales with the number of chunks or basis functions.\n\n[*1] Wang, Peng, et al. \"Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution.\" arXiv preprint arXiv:2409.12191 (2024).\n\n[*2] Zhang, Yuanhan, et al. \"Video instruction tuning with synthetic data.\" arXiv preprint arXiv:2410.02713 (2024).\n\n[*3] Cheng, Zesen, et al. \"Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms.\" arXiv preprint arXiv:2406.07476 (2024).", "questions_for_authors": "- Would segmenting the video by scene change (rather than fixed-length chunks) further boost performance?\n- How are the hyperparameters chosen? For example, is the method sensitive to memory contraction factor? \n- How sensitive are results to the choice of chunk size vs. number of chunks? Is there an optimal chunk length for typical tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6bkjSV726s", "reviewer": "Reviewer_SJwR", "summary": "This paper introduces a long-term memory (LTM) consolidation mechanism from $\\infty$-Former, and a long-video LLM (Language Model) that requires no additional training based on existing short-video LLMs. Experimental results show that this approach significantly improves performance on long-video benchmarks. Furthermore, the use of Gibbs density in LTM for improved PDF sampling, the establishment of continuous-time memory and the use of stick memory sampling strategy are also introduced. The ablation study shows the effects of the proposed sticky memory and LTM. The contribution to understanding brain memory is also highlighted.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and suggestions. We are happy that you found our idea novel, the generalizability from short video to long video understanding well proved in both methods and experiments, and the ablation studies well conducted. We understand your main concerns about our paper and address them below. We hope that our answers clarify and alleviate your concerns.\n\n> “The authors claim that the proposed PDF based on Gibbs density is much powerful than the Gaussian model in L96-99, which lacks neither theoretical proof nor experimental results.”\n\nWe agree that our claim will benefit from concrete evidence. In fact, at an initial stage of our project, we experimented with the Gaussian model but we abandoned it since its unimodal nature limits its ability to capture complex distributions, leading to poor results. Our method extends Video Q-Former's cross-attention using the same learned projections trained with softmax, making Gibbs density a more natural fit. Nevertheless, we conducted additional experiments comparing both approaches, confirming that Gibbs improves performance. We also observed that most of the responses in open-ended MovieChat-1K degraded significantly under the Gaussian model. We will add this to the appendix.\n\n|Density|Method|Acc|Score|CI|DO|CU|\n|-|-|-|-|-|-|-|\n|Gibbs|Ours (no LTM)|68.0|3.76|3.72|3.33|3.71|\n|Gibbs|Ours (unif.)|66.5|3.69|3.60|3.31|3.58|\n|Gibbs|Ours (sticky)|**72.2**|**3.88**|**3.89**|**3.47**|**3.79**|\n|Gibbs|Ours (no STM unif.)|62.4|3.75|3.36|3.38|3.52|\n|Gibbs|Ours (no STM sticky)|59.2|3.68|3.30|3.30|3.44|\n|Gaussian|Ours (unif.)|46.3|2.98|3.60|2.31|3.16|\n|Gaussian|Ours (sticky)|38.4|2.76|3.51|2.11|3.12|\n|Gaussian |Ours (no STM unif.)|46.9|3.06|3.54|2.40|3.13|\n|Gaussian |Ours (no STM sticky)|38.6|2.77|3.43|2.15|3.13|\n\n> “Though the method is training-free, there is still computational overhead should be considered.”\n\nWe refer to our response to reviewer iEVz, where we present additional experiments analyzing the computational overhead, including the time consumption of the LTM module, the impact of sampling more frames, and the encoding time with the ViT encoder. Our results demonstrate that our method sustains a constant memory footprint regardless of the number of frames, with only a slight increase in inference time. Additionally, we observe that increasing the number of basis functions leads to a much smaller rise in memory usage compared to the baseline's growth with additional frames. We hope that these experiments alleviate your concern. \n\n> “The proposed method heavily relies on the video Q-Former. However, recent models seldom use Q-Former (...)”; “The baseline models used for comparison are too outdated (...)”\nWe chose Q-Former because it allows for seamless, training-free integration of our biologically inspired continuous attention component. This is possible because the video Q-Former is designed to process sequences of frame representations using a cross-attention module, which enables our method to augment it with a LTM based on continuous cross-attention. However, our LTM, if we want to go beyond training-free, is not inherently tied to Q-Former and could be used with other architectures. Our primary goal is to provide a proof of concept  incorporating a biologically inspired LTM in a fair, training-free setting. While SOTA models continue to evolve rapidly, our focus was on conducting an apples-to-apples comparison to validate the benefits of our approach. Our method is highly general and can be integrated into stronger VLMs in the future. Unlike our training-free approach, this would require additional learning. We will provide insights in the discussion section. Future work can then build on our findings by incorporating the LTM into the strongest VLMs available.\n\n> “Tab. 1 and Tab. 2 demonstrate limited improvement for VideoChat2 (...). Recent models have significantly outperformed VideoChat2 (...) give more detailed analysis on the reason of limited improvement (...)”\n\nThis is an interesting question. The results in Tab. 1-2 are for multiple-choice datasets, where VideoChat2 models directly incorporate answer options into the prompt, making responses more constrained. In contrast, models like Video-LLaMA generate open-ended responses before selecting the closest option match with LangChain, as explained in the Appendix, creating more room for the LTM to improve reasoning. Nevertheless, in open-ended generation tasks (Tab. 3), our method already provides a significant accuracy boost for VideoChat2 models.\n\nWe are fixing the typos, thank you for pointing them out!", "claims_and_evidence": "Strengths: \n\n1. The introduction of LTM has effectively improved the performance of long video understanding, which has been strongly validated in the experiment section.\n2. The claim of the generalizability from short video to long video understanding is well proved in methods and experiments. \n\n\nWeaknesses: \n\n1. The authors claim that the proposed PDF based on Gibbs density is much powerful than the Gaussian model in L96-99, which lacks neither theoretical proof nor experimental results.", "methods_and_evaluation_criteria": "Strengths:\n\n1. The idea of introducing the brain's working mechanism into video LLMs through structural  design rather than training another model is novel and reasonable.\n2. The use of representing L signals through N basis functions significantly reduces context length, which contributes to longer video understanding.\n\n\n\nWeaknesses: \n\n1. The proposed method heavily relies on the video Q-Former. However, recent models seldom use Q-Former, as previous research [Tong'24] has demonstrated the flaws in Q-Former. Even the latest works of Video-LLaMA series [Zhang'25] and VideoChat series [Li'25] mentioned in the paper, though as the concurrent works, do not use Q-Former. Over-reliance on Q-Former may affect the method's generalizability and limit the possibilities of applying the method to stronger baselines.\n\n\n\\[Tong'24\\]: Tong et. al, Cambrian-1: A Fully Open,Vision-Centric Exploration of Multimodal LLMs, NeurIPS 2024\n\n\\[Li'25\\]: Li et. al, VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling\n\n\\[Zhang'25\\]: Zhang et. al, VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding", "theoretical_claims": "The theoretical introduction of continuous LTM is well presented.", "experimental_designs_or_analyses": "Strengths:\n\n1. The ablation study on different setting is well conducted, proving the effects of each proposed module.\n2. The visualization of the LTM attention density clearly shows the importance of sticky memory. \n\n\n\nWeaknesses: \n\n1. Though the method is training-free, there is still computational overhead should be considered. That includes the time consumption of the proposed LTM module, the extra time brought by sampling more frames from the video and encoding them with ViT encoder. \n2. The experimental results shown in Tab. 1 and Tab. 2 demonstrate limited improvement for VideoChat2, and the authors attribute this to the optimization of the original model. Recent models have significantly outperformed VideoChat2 on various video understanding benchmarks, raising concerns about the applicability of the proposed method on stronger models.\n3. The baseline models used for comparison are too outdated. Most of these models were produced in early 2024 or in 2023, and their performance is far inferior to current models.", "relation_to_broader_scientific_literature": "This paper utilizes the continuous attention [Martins'20] and $\\infty$-former [Martins'22] inspired from cognitive and mechanistic theories of memory [Hardt'09, Weilbächer&Gluth'13, Ma'14] to enhance video models based on VideoChat2 [Li'23] and Video-LLaMA [Zhang'23]. The paper connects the field of neuroscience and video LLM and enables new understanding of neuroscience through the analysis on the mechanism of video LLM memory. \n\n\\[Martins'20\\]: Martins et. al, Sparse Continuous Distributions and Fenchel-Young Losses, JMLR 2022\n\n\\[Martins'22\\]: Martins et. al, $\\infin$-former: Infinite Memory Transformer, ACL 2022\n\n\\[Hardt'09\\]: Hardt et. al, A Bridge Over Troubled Water: Reconsolidation as a Link Between Cognitive and Neuroscientific Memory Research Traditions, Annual Review of Psychology\n\n\\[Weilbächer&Gluth'13\\]: Weilbächer and Gluth, The Interplay of Hippocampus and Ventromedial Prefrontal Cortex in Memory-Based Decision Making, Current Biology\n\n\\[Ma'14\\]: Ma et. al, Changing concepts of working memory, Nature Neuroscience\n\n\\[Li'23\\]: Li et. al, MVBench: A Comprehensive Multi-modal Video Understanding Benchmark, CVPR 2024\n\n\\[Zhang'23\\]: Zhang et. al, Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding, EMNLP 2023 Demo", "essential_references_not_discussed": "Most related works are well cited.", "other_strengths_and_weaknesses": "Strengths: \n\n1. The creative combination and analysis of video understanding and brain memory may contribute to neuroscience\n\n\n\nWeaknesses: \n\n1. The limited performance of the model might be a severe bottleneck for real-world application of the method.", "other_comments_or_suggestions": "1. The last 'a' of Video-LLaMA in L288 should be capitalized.\n\n2. The reference format should be unified and refined. Most names that should be capitalized are not capitalized properly. Some of the references are cited with url while others are not. Some arxiv papers are cited with arxiv IDs, some are cited with url and others are cited without additional information. Some conference names are annotated with abbreviations, while others are not.", "questions_for_authors": "1. Could the authors give proof of the advantage of using the complex Gibbs density for the PDF instead of the original Gaussian model? \n2. Considering the shortcomings of Q-Former, could the authors talk about the generalizability to other models without video Q-Former? \n3. To better prove the effectiveness of the proposed method, could the authors provide a detailed analysis of time consumption for inferring, including the time used for pre-calculating the ridge regression $F^T(F F^T+\\lambda I)^{-1}$, the extra time brought by the use of basis functions $\\psi$, the extra time of using LTM, and most importantly the extra time brought by sampling much more frames and encoding them with ViT encoder? It's recommended to compare the time consumption with LLM forward time. \n4. Could the authors give more detailed analysis on the reason of limited improvement on the strong baseline VideoChat2? \n5. Could the authors conduct the experiments based on stronger baselines to prove the generalizability of the proposed method and provide more comparisons to recent stronger models? \n6. Could the author fix the typos and reference format issues mentioned above?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JpkcGVhAWU", "reviewer": "Reviewer_aCqS", "summary": "The paper presents a method for long video understanding through a continuous long-term memory (LTM) consolidation mechanism. In their approach, the authors propose a continuous-time attention mechanism that leverages the Gibbs density function to obtain a continuous-time query-key similarity function. Using this similarity function, they update the video Q-former's attention mechanism to make it utilize continuous-time LTM, with particular attention on the most relevant parts of input videos given the input prompts, for efficient understanding of long videos. The authors perform relevant experiments to evaluate their approach on multiple video understanding datasets and compare against baseline methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and suggestions. We address your concerns about our paper below.\n\n> “As the authors mention, one key benefit of using basis functions over processing individual video frames is that fewer basis functions are needed to represent the information in the raw frames. This leads to more compressed representations (Lines 145-147), which is particularly useful for processing long videos. However, in their experiments, the authors use 256 frames in each chunk and 1024 basis functions (Line 238, col 2) for VideoLLaMA and 16-frame chunks and 256 basis functions for VideoChat2 (Line 241, col 2). How do these experimental designs reconcile with the assumption of needing fewer basis functions than video frames? Or are the authors implying that the number of basis functions is fewer than the total number of frames across all the chunks (e.g., 8x256 = 2048 > 1024)? But even with that assumption, for VideoChat2, we get 8x16 = 128 < 256. Could the authors please explain this discrepancy?”\n\nFor Video LLaMA, where the number of total frames is large (2048), the basis functions were set to half this total number of frames, ensuring a compressed yet effective representation. In the case of VideoChat2, we used 256 basis functions with $L=128$ frames since the number of frames supported by VideoChat2 is much smaller. Increasing the number of basis functions is computationally lighter than increasing chunk size (as shown in the first two Tables in **iEVz** response) and it improves the multivariate step regression fit. Larger chunk sizes caused memory issues for VideoChat2, but increasing basis functions did not, allowing for better fitting. This strategy balances memory efficiency and model accuracy without overloading computational resources. This tradeoff will be further discussed in the appendix.\n\n> “For the choice of the weighting factor alpha between short- and long-term memories (Eqn. 16), the authors report experiments on the MovieChat dataset and choose the best value of α from those experiments. However, it is unclear how generalizable that value is to other videos or datasets. Also, is it fair to assume that α should be constant for all kinds of videos, or is there scope to make α context- and category-aware, e.g., depending on how often shots are cut in the video, how often background scenes and foreground objects appear and reappear in the video, etc.?”\n\nThank you for the insightful suggestion. While we intentionally used a fixed $\\alpha$ to establish a baseline and simplify experiments, we agree that an adaptive approach—adjusting $\\alpha$ based on video characteristics like shot changes or object permanence—might be an interesting idea to improve generalization in future work. We will mention it in the discussion section.\n\n> “Have the authors considered the effect of any type of noise when determining relevant locations for the sticky memory? For example, if the signal is slightly perturbed (additive noise), how would the probability function (Eqn. 15) be affected? Understanding the noise characteristics is not essential to determine the utility of the sticky memory procedure, and my rating would not depend on the authors' response to this question. However, the noise characteristics become relevant when considering that video signals can easily be corrupted by transmission, compression, or even adversarial noises.”\n\nThis is an interesting question. The impact will depend on the type of noise and on its smoothness in the time dimension. If the signal is perturbed with additive noise, the Gibbs transformation in Eq. 10 will lead to a modified density, which in turn will change the locations for the sticky memory as for Eq. 15. Since the Gibbs transformation is continuous and well-behaved, the effect in the memory representations should be smooth. However, to better understand the practical effect of noise in the memory a careful empirical analysis needs to be conducted. \n\n> “In Eqn. 2, $Q \\in \\mathbb{R}^{R \\times d}$, $K \\in \\mathbb{R}^{L \\times d}$, $V \\in \\mathbb{R}^{L \\times d}$ (...) implies that $Z \\in \\mathbb{R}^{R \\times d}$, but the authors have stated $Z \\in \\mathbb{R}^{L \\times d}$ in both Lines 116 and 121. Is this a typo?”\n\nYes, this is indeed a typographical error. We appreciate your careful review and will correct it accordingly. Thank you for bringing this to our attention.", "claims_and_evidence": "The central claim of chain-of-shot prompting improving video understanding performance is backed up by experimental results.", "methods_and_evaluation_criteria": "The proposed ideas of processing long-term memory in a continuous fashion to improve the attention mechanism of video Q-formers for long videos and sampling relevant parts of the memory at higher granularity to improve Q&A performance make sense as an approach for making long video understanding feasible and efficient.", "theoretical_claims": "Not applicable - the paper presents experimental findings to justify the proposed approach.", "experimental_designs_or_analyses": "1. As the authors mention, one key benefit of using basis functions over processing individual video frames is that fewer basis functions are needed to represent the information in the raw frames. This leads to more compressed representations (Lines 145-147), which is particularly useful for processing long videos. However, in their experiments, the authors use 256 frames in each chunk and 1024 basis functions (Line 238, col 2) for VideoLLaMA and 16-frame chunks and 256 basis functions for VideoChat2 (Line 241, col 2). How do these experimental designs reconcile with the assumption of needing fewer basis functions than video frames? Or are the authors implying that the number of basis functions is fewer than the *total* number of frames across all the chunks (e.g., 8 $\\times$ 256 = 2048 > 1024)? But even with that assumption, for VideoChat2, we get 8 $\\times$ 16 = 128 < 256. Could the authors please explain this discrepancy?\n\n2. For the choice of the weighting factor $\\alpha$ between short- and long-term memories (Eqn. 16), the authors report experiments on the MovieChat dataset and choose the best value of $\\alpha$ from those experiments. However, it is unclear how generalizable that value is to other videos or datasets. Also, is it fair to assume that $\\alpha$ should be constant for all kinds of videos, or is there scope to make $\\alpha$ context- and category-aware, e.g., depending on how often shots are cut in the video, how often background scenes and foreground objects appear and reappear in the video, etc.?\n\n3. Have the authors considered the effect of any type of noise when determining relevant locations for the sticky memory? For example, if the signal is slightly perturbed (additive noise), how would the probability function (Eqn. 15) be affected? Understanding the noise characteristics is not essential to determine the utility of the sticky memory procedure, and my rating would not depend on the authors' response to this question. However, the noise characteristics become relevant when considering that video signals can easily be corrupted by transmission, compression, or even adversarial noises.", "relation_to_broader_scientific_literature": "With the development of language models for video understanding, the paper's contribution is relevant and timely, and it establishes new baselines for future video understanding models. It will likely interest the broader scientific communities working on language models, video understanding, and their intersections.", "essential_references_not_discussed": "While not a domain expert, I did not find any major missing references in my search.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "In Eqn. 2, $Q \\in \\mathbb{R}^{R \\times d}, K \\in \\mathbb{R}^{L \\times d}, V \\in \\mathbb{R}^{L \\times d} \\Rightarrow QK^\\top \\in \\mathbb{R}^{R \\times L} \\Rightarrow QK^{\\top}V \\in \\mathbb{R}^{R \\times d}$. This implies that $Z \\in \\mathbb{R}^{R \\times d}$, but the authors have stated $Z \\in \\mathbb{R}^{L \\times d}$ in both Lines 116 and 121. Is this a typo?", "questions_for_authors": "Please refer to the comments in previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xm0C6DPqws", "reviewer": "Reviewer_j2dG", "summary": "This paper proposes a training-free approach for long-form video understanding tasks. The method introduces a memory bank that integrates modality projectors (Q-formers), which combine short-term and long-term memory for more efficient video processing. By leveraging this architecture, the framework can handle videos without requiring additional training. Experiments conducted on two different video benchmarks, Video-LLama and VideoChat2, show that the method provides clear improvements in long-form video understanding tasks, particularly in video question-answering.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive review and suggestions. We are happy that you found our method to be both efficient and effective for long-form video understanding, and our experiments solid. We address your concerns about our paper below.\n\n> “Is there any efficiency analysis provided in terms of computational resources, such as FLOPs or GPU memory cost for the proposed method? For example, a visualization like the figure 1 in MovieChat paper would be better.”\n\nWe appreciate the suggestion and refer to our response to reviewer iEVz, where we present additional experiments that analyze the computational overhead. These experiments include the time consumption of the LTM module and the impact of sampling more frames. Our results demonstrate that our method sustains a constant memory footprint regardless of the number of frames, with only a slight increase in inference time. Additionally, we observe that increasing the number of basis functions leads to a much smaller rise in memory usage compared to the baseline's growth with additional frames. We will add these experiments to the appendix.\n\n> “In Table 3, why do video-llama-based models perform best in sticky settings while VideoChat2 are in no short-term memory sticky?”\n\nThis is indeed intriguing and we asked ourselves the same question. We speculate that VideoChat2-based models tend to prioritize global features, whereas Video-LLaMA tends to focus on local features. Both benefit from sticky memories in open-ended generation tasks, as it helps maintain long-term coherence and consistency across outputs, regardless of whether the model emphasizes global or local context.\n\n> “Is it possible to set alpha as a dynamic parameter instead of keeping it fixed at 0.9/1.0 statically? A dynamic alpha might allow more flexible adaptation based on varying video contexts.”\n\nThank you for the thoughtful suggestion. We agree that using a fixed $\\alpha$ may not always be ideal, as video content with different temporal dynamics might benefit from varying weightings. While we chose a fixed $\\alpha$ to simplify the current experiments and establish a baseline, we recognize the potential advantages of making $\\alpha$ dynamic. Adapting $\\alpha$ based on video characteristics (as mentioned by reviewer **aCqS**), such as shot transitions or object persistence, could indeed improve performance and generalization. We plan to explore this dynamic approach in future work, and we will mention it in the discussion section.\n\n> “The baseline models used in the paper (Video-LLama and VideoChat2) are quite old (from spring and summer 2023). Given the rapid advancements in video understanding, there are newer and more powerful models available. It would be better to consider including some of these models, such as Llava-OneVision[1], Qwen2-VL[2], or DeepSeek-VL[3], in future experiments?”\n\nWhile we recognize the rapid advancements in video understanding and the availability of more powerful models, our primary goal in this work was not to develop a SOTA model but to demonstrate the effectiveness of incorporating a biologically inspired LTM in a fair, training-free setting. The baseline models we used (Video-LLama and VideoChat2) were chosen for their relevance to our approach given that their Q-former based architecture enables a training-free approach, but we agree that including newer models like Llava-OneVision, Qwen2-VL, or DeepSeek-VL in future experiments (which would require additional training) may provide valuable comparisons. Our method is highly general and can be integrated into stronger VLMs moving forward. We will highlight this in the discussion section and suggest future work to explore the incorporation of our LTM approach into state-of-the-art models.", "claims_and_evidence": "Yes, the claims made in the submission are supported by clear and convincing evidence from the experimental results. The authors demonstrate that their approach, which integrates a memory bank and modality projectors, effectively improves the performance of video understanding models on standard benchmarks.", "methods_and_evaluation_criteria": "Yes, the proposed training-free method has the impact on application side, addressing the challenge of long-form video understanding without requiring additional training.", "theoretical_claims": "There is quite a little theoretical part in this paper. The authors primarily focus on the empirical performance of their method, with limited theoretical exploration of how the continuous-time memory mechanism interacts with the video understanding tasks.", "experimental_designs_or_analyses": "The experimental settings are clear. The paper presents well-defined experiments with appropriate baselines and evaluation metrics. However, one concern is that the authors seem to use relatively old models as the base for their comparisons.", "relation_to_broader_scientific_literature": "The proposed method further extends the memory-based long-form video understanding framework by incorporating more dynamic resource allocation. By augmenting the memory model with a continuous-time mechanism, the approach provides a way to process long-form videos more effectively.", "essential_references_not_discussed": "The paper ignored some recent state-of-the-art video-language models, such as Apollo[1], Video-LLaMA[2], and Llava-OneVision[3]. These models have made significant contributions to the field and could provide useful comparisons for the proposed method.\n\n[1] Zohar, Orr, et al. \"Apollo: An exploration of video understanding in large multimodal models.\" arXiv preprint arXiv:2412.10360 (2024).\n\n[2] Zhang, Hang, Xin Li, and Lidong Bing. \"Video-llama: An instruction-tuned audio-visual language model for video understanding.\" arXiv preprint arXiv:2306.02858 (2023).\n\n[3] Li, Bo, et al. \"Llava-onevision: Easy visual task transfer.\" arXiv preprint arXiv:2408.03326 (2024).", "other_strengths_and_weaknesses": "Strengths:\n1. The proposed method uses a memory bank to address long context has been proven both efficient and effective for long-form video understanding. \n2. The paper extended the framework from MovieChat by introducing Continuous-Time Memory Consolidation, which is better than uniformed sampling. The continuous-time strategy helps capture more relevant video segments and improves overall efficiency.\n3. The experiments are solid, utilizing multiple widely-used datasets, which makes the results more robust and generalizable to various tasks in video understanding.\n\n\nWeaknesses: See Question Section.", "other_comments_or_suggestions": "NA", "questions_for_authors": "1. Is there any efficiency analysis provided in terms of computational resources, such as FLOPs or GPU memory cost for the proposed method? For example, a visualization like the figure 1 in MovieChat paper would be better.\n2. In Table 3, why do video-llama-based models perform best in sticky settings while VideoChat2  are in no short-term memory sticky? \n3. Is it possible to set alpha as a dynamic parameter instead of keeping it fixed at 0.9/1.0 statically? A dynamic alpha might allow more flexible adaptation based on varying video contexts.\n4. The baseline models used in the paper (Video-LLama and VideoChat2) are quite old (from spring and summer 2023). Given the rapid advancements in video understanding, there are newer and more powerful models available. It would be better to consider including some of these models, such as Llava-OneVision[1], Qwen2-VL[2], or DeepSeek-VL[3], in future experiments?\n\n[1] Li, Bo, et al. \"Llava-onevision: Easy visual task transfer.\" arXiv preprint arXiv:2408.03326 (2024).\n\n[2] Wang, Peng, et al. \"Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution.\" arXiv preprint arXiv:2409.12191 (2024).\n\n[3] Lu, Haoyu, et al. \"Deepseek-vl: towards real-world vision-language understanding.\" arXiv preprint arXiv:2403.05525 (2024).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: While the paper has some weaknesses related to newer model comparisons and computational efficiency analysis, it introduces a highly novel and impactful method for long-video understanding without requiring training. The method is well-supported by experimental results, showing solid improvements across various video QA benchmarks. Given the paper's novelty, experimental soundness, and strong theoretical foundation, the recommendation is to accept. Further improvements could include comparing the method with newer models and a more thorough analysis of computational overhead for very large videos.
Decision: Accept (poster)