Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge
Authors: Haomiao Xiong
Zongxin Yang
Jiazuo Yu
Yunzhi Zhuge
Lu Zhang
Jiawen Zhu
Huchuan Lu
Venue: ICLR 2025 Poster (2025)
Keywords: Streaming Video Understanding; Video MLLM; Hierarchical Memory System
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: A Vertasile Approach and Benchmark for Streaming Video Understanding and Multi-round Interaction

Abstract: Recent advances in Large Language Models (LLMs) have enabled the development of Video-LLMs, advancing multimodal learning by bridging video data with language tasks. However, current video understanding models struggle with processing long video sequences, supporting multi-turn dialogues, and adapting to real-world dynamic scenarios. To address these issues, we propose StreamChat, a training-free framework for streaming video reasoning and conversational interaction. StreamChat leverages a novel hierarchical memory system to efficiently process and compress video features over extended sequences, enabling real-time, multi-turn dialogue. Our framework incorporates a parallel system scheduling strategy that enhances processing speed and reduces latency, ensuring robust performance in real-world applications. Furthermore, we introduce StreamBench, a versatile benchmark that evaluates streaming video understanding across diverse media types and interactive scenarios, including multi-turn interactions and complex reasoning tasks.  Extensive evaluations on StreamBench and other public benchmarks demonstrate that  StreamChat significantly outperforms existing
state-of-the-art models in terms of accuracy and response times, confirming its effectiveness for streaming video understanding. Code is available at StreamChat.

Published as a conference paper at ICLR 2025
STREAMING VIDEO UNDERSTANDING AND MULTI- ROUND INTERACTION WITH MEMORY-ENHANCED KNOWLEDGE
Haomiao Xiong'; Zongxin Yang"; Jiazuo Yu', Yunzhi Zhuge'! Lu Zhang', Jiawen Zhu', Huchuan Lu
Dalian University of Technology, Harvard University
ABSTRACT
Recent advances in Large Language Models (LLMs) have enabled the develop- ment of Video-LLMs, advancing multimodal learning by bridging video data with language tasks. However, current video understanding models struggle with pro- cessing long video sequences, supporting multi-turn dialogues, and adapting real-world dynamic scenarios. To address these issues, we propose STREAMCHRT, training-free framework for streaming video reasoning and conversational inter- action. STREAMCHAT leverages a novel hierarchical memory system to efficiently process and compress video features over extended sequences, enabling real-time multi-turn dialogue. Our framework incorporates parallel system scheduling strategy that enhances processing speed and reduces latency, ensuring robust per- formance in real-world applications. Furthermore, we introduce STREAMBRe BTRNC, a versatile benchmark that evaluates streaming video understanding across di- verse media types and interactive scenarios, including multi-turn interactions and complex reasoning tasks. Extensive evaluations on STREAMBENCH and other pub- lic benchmarks demonstrate that STREAMCHAT significantly outperforms existing state-of-the-art models in terms of accuracy and response times, confirming its effectiveness for streaming video understanding. Code is available StreamChat.
INTRODUCTION
Recent advancements in Large Language Models (LLMs) [1-3] have led to the development Video- LLMs [4-9], which aim interpret visual scenes, ac- tions, and narratives. These models represent significant progress in multimodal learning by bridging video data and language-bars tasks, with applications spanning from content analysis to human-robot interaction [9].
43.,4%
Object Senrch 50.5%
NEAT-QA 50.5%
Long-term Mcmory
Activity 50.1%
Despite these advancements, current offline models primarily process videos static clips and rely single-turn dialogues, incorporating visual information through mechanisms like projection layers 8] or cross-attention structures [2]. However, these models encounter computational bottlenecks when handling ex-
Short-term Menury %6.1%
Simple Start 68.8%
Interaction 67.6%
Konwlelge
VidesCnline SIB Flac--Soccnn-7
VID-A-
FreeVA-7 SucamnChut-7
LLV--LLVV-T
Figure 1: Performance comparison between STREAMCHA' and previous Video-II lengthy video features within limited memory resources [10]. Additionally, their inability to support multi-turn dialogues reduces adaptability for interactive scenarios, and key information may be lost due to insufficient video sampling methods (ef. Fig. 2(a)).
tended video sequences, often struggling to compress
To address these issues, online models [10, 11] have emerged. They utilize memory-based approaches and temporally aligned instruction-tuning to process long videos and enable multi-round interactions (cf. Fig. 2(b)). While these models allow dynamic user interactions, they still face challenges
"Equal contribution.
1 "Correspnniigg author begyz@dlut.edd

---

Published as a conference paper at ICLR 2025
323
66s
105x
124
153
1.9
reitec sxad Aooer rocir to ashanee shtr
Senty corrt any sustr
Ansugle w.li age
Whar shing??
oud
VNSAR rood your Iwaard Con
Deogght valk
What the dloce
Fuutt Fanrd Jonr soild etblanc touns Can Aach
Aee Nand jusr
voliene shat Shere Fooor Is my roue??
Singlo Hennaa Video
poicr of offensd win Syph Good
video is buildng
Slow Procem
BERBON
mosch noun?
Sonc
shore white boccee
velere zo food sheae
X
Ie gresn hooill
oppaare sorting
Mallcorani Dart Prpcess Crneet Prnwer
of wemeeaherr coenentyy thy chouss
mash or Macil hanalle
(n) Existing Offline Method
(b) Existing Online Method
(c) Our Method
Figure 2: The comparisons between Stream Chat and other methods (81). Offline methods process entire
videos, leading to information loss and limited to a single interaction. Previous online methods [10, 11J enable multi-round interactions but still suffer from slow processing and answer correctly. The proposed method achieves real-time video processing, improving the efficiency and accuracy with memory support.
maintaining rapid processing speeds and consistent performance across unfamiliar scenarios critical factors in real-time applications like robotic navigation and human-robot collaboration.
To overcome these limitations, we propose STREAMCHAT. a training-free framework for streaming video understanding that offers three key innovations: (i) Training-free adaptability, allowing to efficiently process videos of various types and lengths without sesourceeheavy training. This makes STREAMCHAT suitable for both online and offline video processing while maintaining stable performance across diverse scenarios. (ii) Hierarchical memory storage, which manages and compresses video information over long sequences. It integrates short-term memory for tracking ongoing events, long-term memory for retaining past events in compressed form, and dialogue memory to maintain conversational history, ensuring continuous and coherent dialogue understanding. (iii) Optimized system scheduling, which improves model inference efficiency by processing tasks in parallel across three threads: the selective frame stacking thread identifies and removes redundant frames, the memory formation thread updates and refines memory information, and the contextual summarization thread handles user requests and generates responses in real-time.
We evaluate STREAMCHAT on existing benchmarks [12-16] and identify their two major shortcomings: (i) Short and monotonous video content, which fails to capture the complexity of real-world streaming media, and (ii) Simplistic, single-round questions, which do not test the model's ability to engage in multi-turn dialogue or complex reasoning.
To address these deficiencies, we introduce STREAMBENCH, a comprehensive benchmark designed for streaming video understanding. It includes a diverse array of video content such as egocentric videos, web videos, and movie scenes, paired with text annotations that simulate multi-round interactions. In terms of video selection, we perform rigorous manual curation from large datasets to ensure both high-quality content and a broad range of categories. distinct types of queries, probing various dimensions of the model's I' reasoning abilities, from simple factual retrieval to complex inference. Compared to previous benchmarks, STREAMBENCE not only evaluates the accuracy of model responses but also incorporates latency metrics, which essential for assessing performance in real-time applications. This comprehensive evaluation framework offers a more realistic and reliable measure of model robustness and practical utility.
terms of quextions, we design six
In summary, our key contributions are as follows:
We propose STREAMCHAT, a training- frrei method for streaming video understanding. Its novel hierarchical memory storage and system scheduling strategy enables robust memory management, real-time video processing, and multi-round interaction capabilities. These features ensure precise and efficient response generation, catering to the dynamic nature of video contexts.
We introduce STREAMBENCH, the first comprehensive benchmark to evaluate streaming video understanding models. This benchmark simulates real-world interactions through multi-turn dialogues and diverse question formats, offering detailed assessment of model performance.
STREAMChAT sets new benchmarks (ef. Fig. 1), delivering 64.7% accuracy STREAMBENCH for online settings, which is an 8.3% improvement over the previous best. offline scenarios, outperforms the ofte-oo-the-ar method by average of 2.5% across four public benchmarks.
2

---

Published as a conference paper at ICLR 2025
Lengt Dirtibction
Phane
Cooking
Madaun
Menoloarlom
Contrnccton
Mivic
Wooksn
(19t pag
(199 tag.
(188 Stay
can zey
2ng.
Cre sky
488 Lopp
ase Aoy
Gedi suy
cog
(49 409
ratt eny
(eeg Stay
ara s
Oundeur
Room: Your
Quemion Dienbatin
Livigrones
Web Video 37%
Nunter of Questass
303
Xodiay
Orters
Closroom
Bethooum
Cumady
150
Showe
Search
Loop una Monert
Ston uno Monint
Uidiose Intetctionn
Koneledge laned
Soggle Stort
Figure 3: Benchmark overview ($2). Our benchmark covers 4 key domains and 16 sub-class video types. These videos exhibit a broader distribution of length, with 6 different types that are evenly distributed.
In terms of efficiency, STREAMCHAT achieves a processing speed of 32 FPS, marking sixfold increase over existing methods. Additionally, maintains text generation lateney under 0.9 seconds, showcasing significant advances in interactive video processing.
2
COLLECTION AND COMPOSITION OF STREAMBENCH
2.1 VIDEO COLLECTION
Previous video understanding benchmarks [12-14, 16- 18] primarily focus on offline scenarios, where all video frames and user questions are provided to the model simul- taneously for generating answers. To find more suitable method to assess a model's 's ability to understand online sce- narios, we introduce STREAMBENCH, a benchmark mainly designed to simulate online video scenarios. There are two distinct differences compared with other video un- derstanding benchmarks. (i) Diverse video curation: we collect four major domains and sixteen sub-classes of video sources, including egocentric videos, web videos, working videos, and movies the database of the benchmark. Each type has its unique characteristics and challenges, which can verify the stability and reliability of the model in a wide range of application scenarios. (ii) Crafted query types: we design six types of questions to meet the specific needs of online video understanding and ensure that these types of questions appear once in single video, forming multi-roume dialogue. This section introduces how we collect videos and construct annotations. More details about the diversity and distribution are shown in Fig. 3.
Table 1: Comparisons different bench- marks. MRT denotes multi-round interac- tions.
Benchmark
MR
Avg
Total
Video
QA
MSVD [16]
X
10s
1.4h
Weh
Desc.
MSRVTT [12]
X
15s
12.5h
Web
Dese.
ActivityNet [14]
X
112s
Web
Dese.
Next-QA [15]
X
40s
11h
Web
Temporal
MovieChat [7]
Movie
Movie
STREAMBENCH
270s
25h
Mix
Online
Data Sources. In selecting videos for benchmarks, we prioritized diversity in type and length to maintain high data quality. Our primary sources are the EgoSchema [13] and YouTube--M [19] datasets. EgoSchema offers a rich array of both indoor and outdoor scenes, providing an extensive range of egocentric perspectives and actions, which aligns perfectly with our experimental needs. From YouTube-SM, which features comprehensive internet-sourced collection spanning over 4,000 classes, we filtered to procure high-qualit web, work-related, and cinematic videos. This diverse selection framework ensures our model is tested against a broad spectrum of real-world scenarios.
Filtering videos. It is a crucial step to ensure the quality of the videos used in the benchmark. To achieve this, our data filtering pipeline consists of machine and human selection. Firstly, a multi- modal language model [20] is utilized to classify the original data. The categories of videos are provided by data source, we feed them with the videos to the machine and make it select the category of the video. Secondly, human judgment is required to assess the redundancy: the change of scenes in videos. Some static video content (e.g., ego view of drawing, watching TV) and high-noise data removed according to human judgment.
from web videos (e.g., video games, advertisements) Finally, StreamBench consists of 306 videos with a total duration of 24.8 hours and an average of 4.5 minutes each, offering comprehensive collection of videos from different categories and lengths.
3

---

Published as a conference paper at ICLR 2025
2.2 CONSTRUCTION OF TASKS AND ANNOTATIONS
We have crafted six distinct tasks with annotations to simulate the conversation between the agent and the human. Each task corresponds to a different real-world scenario, ensuring comprehensive coverage of potential communication contexts.
Object Search (OS): Challenges include accurately describing an object's position in video. The task conditions are that the object must appear for less than seconds and the interval from its appearance the user's request should exceed 30 seconds, enhancing the difficulty of the search. Long-term Memory Search (LM): This task assesses the model's memory by requiring recall of events appearing for more than 5 seconds, with delay exceeding 1 minute from the event's end to the user's query, testing long-term memory retention.
Short-term Memory Search (SM): To simulate the user's interest in recent events, this task sets the interval from event completion to the user's query at less than 20 seconds, evaluating the model's response to recent activities.
Conversational Interaction (CI): Sometimes the answer to a user's current question is closely related to conversation history. Therefore, the model must memorize conversation records and retrieve the most relevant text from the memory contextual support. This type is designed to simulate multi-turn dialogue scenarios. We set the dialogue information associated with the user's current request to come from any previous conversation, with interval of more than 2 dialogues. Knowledge- Question Answering (KG): This type of question evaluates the model's internal knowledge, which is retained by the base large language models. In this benchmark, we set the questions must be related to the events objects occurring in the video so that it can simulate scenarios where users have specific need to understand background or encyclopedic knowledge.
Simple Factual (SF): This type of question focuses on friendly dialogue starting between the user and the model. Therefore, they must be asked within 30s after the beginning of the video. Although the question is simple, the model needs to remember things in the short term to answer correctly.
To ensure the quality of the annotations, we additionally assign different workers to perform human feedback for manual annotation. The human feedback step needs to focus three parts: (1) check the question formats are correct and diverse, (2) ensure the expressions are clear and consistent with the video, and (3) remove sensitive topics such as those questions related to nationality politics. These steps of manual annotation and feedback, along with multi-modal large language model assisted video collection, form semi-automated benchmark construction pipeline (Appen. 5A). Finally, STREAMBENCH contains 1.8K high-quality QA pairs. The distribution of these annotations is shown in Fig. 3. Some examples from the benchmark that offer an intuitive observation of our annotation results and formats are shown Appen. $B.
3 STREAMCHAT
Given streaming broadcast video and timestamped questions as input, STREAMCHAT is designed to efficiently perform reasoning and deliver accurate answers across multiple rounds. Building upon Long VA [20] (Appen. 5F) foundational Video-IILM, our design incorporates two key components: a hierarchical memory storage system ($3.1) that leverages long-term, short- term, and dialogue memories to compress and manage extensive video sequences within constrained resources. thereby facilitat- ing effective
Table 2: Comparisons of recent video- milms. Our streaming (S.) method with memory (M.) achieves processing video real-time (R.) and generates re- sponse with low latency (I..).
M.
Vidco-LLaVA
X
X
X
X
MovieChat [7]
X
/
X
x
reasoning; and system scheduling strategy ($3.2) that decouples video feature extraction from memory updates, thereby preventing unbounded buffer growth as the input video frames increase.
Video-online [[]]
/
X
X
x
Blshh-SStream [10]
/
STREAMCHAT
/
HIERARCHICAL MEMORY STORAGE
STREAMCHAT treats videos as dynamic information repositories, utilizing hierarchical memory to analyze and store the diverse content. This section details two specialized memory structures devised to address the challenges of information storage and retrieval: long-short term memory
4

---

Published as a conference paper at ICLR 2025
Disparate
Congruent
Could you please fell me what happened to in blach aweater's boul?
Vision butfier filling
Wleem
Pop&
Previous
100%
58%
107
Push
Selective Frame Stacking
C Current Inference
thought
question
eetreved Tokcns
According to mensory the while ship fitt watl with white cluuds und blar
Answer
ingut
contest
Herarchical
Memorr
quastiion
Memory
record
record
hissory context
you............
update
retrieval
O&A History
Memory Formation
Contextual Summarization
Figure 4: Overview of StreamChat ($3), which comprises three main components: (i) Selective frame stacking, which prepares vision features for processing. including encoding frames and filling the vision buffer; (ii) Memory formation, where vision features are organized into structured memory; (iii) Contextual summarization, utilizing hierarchical memory to respond to user queries by providing relevant context.
Mt U, = 14.20 Th u {s and dialogue memory M2 = {d+ These memories manage visual and conversational data, where T is video duration, S is short memory length, D counts dialogues, and L is the chunk size for long memory. The following sections introduce the functions of the above parameters.
3.1.1 LONG-SHORT TERM MEMORY
Selective Frame Stacking. To reduce the feature storage overhead caused by redundant frames in videos, we use Lucas-Kanade Optical Flow algorithm [21] in the selective frame stacking module
to assist in determining the validity of each video frame {P' E B x317
J==0 Specifically, we
calculate the motion vector (u, v) between i-th frames F and the last frame Ft11
Σ. - (1)
(1)
E.400
Σ. (1)
where (t), represent the partial derivatives of the frame F with respect to position i(,, y) and time t. We develop the motion vector magnitude |||| v E [0, 1] to represent the total motion intensity between frames. If |O]| exceeds the predefined threshold E [O, 1]. the frame F will be encoded into vision embedding e' E Rn and pushed into buffer Biision-
Short-term Memory. We intend to design human- memory method that simulates the Atkinson- Shiffrin model [22]. which emphasizes the role of a short-term storage for maintaining readily accessible, frequently updated information. Specifically, as shown in Fig. 5 (a), we select N vision embeddings from the Biision as vision candidates C. Building on the Ebbinghaus Forgetting Curve theory [23], we handle memory updates by randomly selecting S vision embeddings e from C to construct the short-term memory Ma:
C (ox-1ef-(N-1) NNN-2e (N- (N- ,..., J0e")
random
M, (s; E RRXd1S
select
where a; is the normalized forgetting probability of i-th unit of represents the length.
Long-term Memory. The long-term memory simulates the complex and abstract memory of humans [22]. For this reason, we design two forms of information in long-term memory: text clues, which is used store declarative text describing events that occurred over past period, and vision memory, which is used to store compressed visual features U; E Cxx Text clues serve as an index for retrieving relevant information from the long-term memory (introduced in $3.1.3). Our system overcomes the bottleneck of VRAM consumption and the challenges of retrieving memory units I; by constructing tree structure shown in Fig. 5 (a).
The construction of the long-term term memory tree can be outlined in the following steps: Firstly, the vision buffer is chunked, and each chunk is clustered and assigned caption:
5

---

Published as a conference paper at ICLR 2025
summanze SUMII
Text i
Long memory tree M1
Short-term memory M, M,
Hierarchical Memory
MILM
lext
Mi
d,
select
further clustering
Fl
io
Weighteed
Ma
dq
text
Wriite
captioming
Ko
K
d:
Ma
video time line
comnersation history
(b) Dialogue Memory Long-short term memory, where the long memory tree Mj and short-term memory M, constructed along the video time line. (b) The dialogue memory Ma updated after each inference conversation for managing the dialogue historics.
(a) Long-short term Memory
Figure 5: The hierarchical memory storage ($3.1). I). (a)
Byision (D K; { o: v. - fk-means (K.), pu(s:
where E Rxdd is the i-th cluster formed from feature chunks (K,) C' is the clustering goals and tę represents the i-th caption of each chunk. Each chunk Ac, contains L features e' E R which come from buffer Biision- Next, clustered feature v; and a caption 4 together form a long memory unit 4, which also serves as the basis nodes of our tree structure:
[Lu. l........ 1...1....44411]pm 1--denodes push 4 = {er, t
Finally, basic nodes are further grouped into higher level nodes [1] 1, ..................... [H in chronological order until a tree structure is formed and all the basic nodes are exhausted
Mi T/4 1 = me po (ir (4))
(5)
3.1.2 DIALOGUE MEMORY
In our approach, each round question {Q and answer {A.) is viewed as a memory fragment, which is pre-encoded by the encoder model E(-) into contextual representation d, Thus, the entire dialogue history M. is pre-encoded as shown in the following formula: Ma
{do, d1, di...., d=-1} push d, = <<< Qi A, >) where the length of M3 is equal to the conversation number D, and select MiniLM-L6 [24]
encoder model.
3.1.3 RETRIEVAL
When user question Q; comes in, the memory system will search for the most relevant knowledge supplementation by the retrieval algorithm. In long memory tree M,, the Qi and text clue units {t.) T/L are encoded by the tokenizer and the embedding layer of LLM. Based on the cosine similarity between encoded Q, and text clue t, the memory system will search for the retrieved tokens M, U {u, E RCxd [F= where L represent the layer number of M,. In dialogue memory Ma- the user requests Q; is encoded by E(-) query to search for the context < Qretreveed Averieved > based on the FAISS [25] index. More details about our retrieval algorithm are shown Appen. SC.
3.2 SYSTEM SCHEDULING
As shown in Fig. 4, our method includes three different parts: selective frame stacking, memory formation and contextual summarization. These components are operated as independent threads to optimize inference speed and minimize latency. System scheduling is crucial as it enables concurrent execution of these threads without interference, significantly enhancing processing speed.
Specifically, the (i) slective frame stacking thread actively populates the vision buffers Bissi with features e Once full, these features are cleared from the buffer and passed to the (ii) memory forma- tion thread, which updates the memory structures by building the long-term memory tree Mr and refreshing the short-term memory Mg. Concurrently, previous dialogue records (< SG-1,A,-1 >) stored in the dialogue memory Mg. Upon receiving
query Q, the (iii) contextual sum- marization thread retrieves relevant information from the hicrarchical memory to provide timely responses. This architecture supports sub-second latency (<0.9s) and video processing up 32 FPS.

---

Published as a conference paper at ICLR 2025
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Memory Configurations. To adapt the model to various application scenarios, we configure three versions with different memory settings: Base, Fast, and Slow. These variants adjust key memory parameters, including threshold (4), chunk length (L.), group size (g), and clustering goals (C), as summarized in Tab. 3. The Fast model is optimized for rapid video processing, while the Slow model prioritizes accuracy in responses. balances processing speed and accuracy.
Table 3: Memory configura-
tions for three models.
Version
g
slow
0.13 35
15
5
Base
10
Fast
0.58 30
15 5
The Base model
Evaluation Metrics. We evaluate semantic similarity in single conversations using the LLaMA-3
model [3], which assigns semantic correctness score (Sco.) ranging from [0, where higher scores reflect responses that more closely align with the expected answers. For assessing coherence in multi-turn dialogues, we compute score fluctuations across turns; smaller fluctuations (Coh.) indicate a smoother dialogue experience. Additionally, we measure request processing delay (RPD), defined as the time (in seconds) from user request submission the start of response generation. A smaller RPD signifies lower latency, resulting in reduced wait times for users. Appen. jD offers more details.
Implementation Details. We utilize CLIP-L-P14 [26] as the vision encoder and we set the number of selected memory units S' to 5 and candidate length C to 20. Experiments were conducted on two NVIDIA Tesla A800 GPUs with 80GB of memory each (more details in Appen. $F ). We benchmark our model against ofte-of-theear methods, including Video-LLaVA [4], LLaMA-VID [2] and etc.
4.2 COMPARISON WITH STATE-OF-THE-ARY METHODS
Online Scenarios. As shown in Tab. 4, our models demonstrate significant improve- ments over the previous best method, Video-
Table 4: Quantitative results in StreamBench. RPD is
measured for streaming methods. Fr.: sampled frames.
Method
EPS
Fr.
Sco.
Acc.
Coh.
RPD
online [11].
Human performance
4.03
79.4
GPT-4o [27]
50
3.70
71.0
1.66
Slow: Achieves an 8.3 % higher accuracy
GPT-4o [27]
3.64
and a 0.37 higher score than Video-online.
GPT-to-mini [27]
35
3.17
59.1
2.01
Instract tuming
Fast: Processes video 32 FPS, making
Video-IL [4]
2.81
48.9
2.19
LLaMA-VID [2]
180
2.94
51.2
2.08
it much faster than all previous streaming methods, while still improving accuracy by 5.3% and scoring 0.17 higher than [1 1].
LLaVA-NEXT
2.65
LLaVA LLaVA-HHoud [29]
3.12
54.7
1,83
LongVA [20]
3.05
MiniCMP-v26
2.97
56.6
2.21
Base: Reaches 63.8% Acc. and 3.42 score.
VII.AL.5 [31]
3.10
57.1
2.20
InternVI.2 [32]
3.15
57.6
2.11
Best model: Surpasses [11] with a 0.18 im- provement in coherence score and reduces latency by 0.17s, delivering smoother con- versations with shorter wait times.
IntenL.MXXCP2.5 (33)
3.21
57.7
2.12
Traiming-free
MovieChat [7]
32
2.07
35.3
2.36
FreeVA [8]
3.10
56.3
2,11
Streuming
Video online [1]]
3.11
56.4
1.94
1.07
Due to system scheduling, all models main- tain nearly the same response time of about 0.9s. Tab. 5 presents the detailed scores across six question types. Using hierarchical memory storage, our method excels in object
Flash- SStrea [10]
2.89
52.1
2.21
4.15
STREANCHAT
3l
15
3.48
64.7
1.76
0.90
Base
3.42
63.8
0.89
Faat
32
3.28
61.7
1.81
0.85
search (OS), long-term memory search (LM), short-term memory search (SM), and conversational interaction (CI) tasks. Notably, our Slow model increases accuracy by 10.3% in OS, 5.1 in LM, 4.9 % in SM, and 5.8% in CI compared Video-online.
Offline Scenarios. We compare our Base model against other methods in general offline video understanding benchmarks including MSRVTT-QA [12], ActivityNet [14], NExT-QA [15], MSVD- QA [16]. Since these benchmarks involve open-ended questions, we evaluate performances using score and accuracy as metrics, employing the same score model [3] used in online tests. It should be noted that considering the limited average video length the MSRVTT [12] and MSVD [16] shown Table we did not apply long-term memory Mi for our model during the test. Additionally, since these open-ended question-answering test format benchmarks do not evaluate multi-round dialogue capabilities, we removed the dialogue memory Ma component from our model.
7

---

Published as a conference paper at ICLR 2025
Table Quantitative comparison across six tasks. Detailed results for tasks 'OS", 'LM', 'SM', 'CT' 'KG', and 'SF". For full names and definitions, refer to $2.1
LM
Method
Publication
Seo.
Acc.
Sco.
Acc.
Sco.
Ace.
Sco.
Acc.
Sco.
Acc.
Sco.
Acc.
Homan performance
71.8
3.81
4.07
4.14
82.6
80.7
4.30
GPT-4o-50 [27]
Arxiv
2024
3.27
60.5
3.35
61.2
3.41
64.4
3.81
72.3
4.58
93.9
3.83
74.7
GPT-4o-35 [27]
Arxiv
2024
3.22
59.6
3.28
58.6
3.45
65.3
3.76
71.7
4.54
93.3
3.50
66.1
GP mini 35
Arxiv
2024
2.52
46.8
2.70
45.8
2.80
51.0
3.50
64.0
4.67
95.2
2.90
53.3
Instract-
Vidoo-LLaVA [4]
EMNLP
2024
2.25
31.2
2.31
2.50
41.8
3.18
56.1
3.81
74.6
2.93
LLaMA-VID [2]
BCCV
2024
2.32
33,9
2.43
38.2
2.63
44.1
3.31
58.4
3,93
76.9
3.06
57.1
[331]
CVPR
2024
2.33
36.1
2.54
44.3
2.87
50.8
3.59
68.3
3.97
78.6
3.38
65.5
InternV1.2
CVPR
2024
2.49
38.5
2.70
46.6
2.89
50.9
3.61
67.6
4.02
81.0
3.29
62.2
LLaVA-NEXT [28]
Artiv
2024
2.17
35.0
2.14
31.4
2.15
36.0
2.55
42.7
3.88
76.1
3.12
$7.6
LLaVA-Hound [29]
Arxiv
37.6
2.68
3.09
53.4
3.21
55.7
76.3
3.35
LongVA [20]
Arxiv
2024
2.61
41.8
2.81
47.4
3.20
57.6
3.29
59.8
4.01
80.7
3.48
66.1
MiniCMP-v2.6
Arxiv
2024
2.32
37.6
2.78
51.9
2.62
43.7
3,35
65.7
3.19
66.2
3.27
64.2
IntemLLM-XCP2.55 1331
Arxiv
2024
2.40
38.8
2.81
43.3
2.89
50.8
3.62
65.6
4.41
88.4
3,23
60.5
Fraining-Free
MovieChat [7]
CVPR
1.42
1.76
42.3
2,05
Free [8]
Arsiv
2024
2.39
35.66
2.33
37.5
2.62
43.7
3,16
58.8
4.24
84.0
2.87
53.7
Onfine
Video-online [1]]
CVPR
2024
2.61
41.4
2.87
48.8
3.01
52.9
3.31
62.7
3.58
69.2
3.39
64.1
Flssh-VStram [10)
Arxiv
2024
2.38
37.1
2.64
44.5
2.78
48.6
3.13
58.1
3.34
66.4
3.17
59.2
STRISANCAA
Slow
3
51.7
2.93
53.9
3.21
57.8
3.86
68.5
4.38
88.1
3.57
69.3
Base
2.93
50.5
2.87
52.9
3.15
56.1
3.82
67.6
4.37
87.9
3,56
68.88
2.78
48.1
2.73
3.02
53.5
3.69
65.2
4.12
86.7
3,46
67.6
Table 6: Performance comparison of various models in offline vidco understanding benchmark.
ActNet
NEXT-QA
MSVD
MSRVTT
Average
Method
Publication
Sco.
Acc.
Sco.
Acc.
Acc.
Sco.
Acc.
Sco.
Acc.
Video LLaVA [4]
EMNLP 2024
1.96
35.8
2.02
34,9
2.94
57.5
2.24
42.8
2.29
42.7
IJaMA-VID
FCCV
2024
2.09
36.6
2.07
36.0
2.83
56.9
2.23
42.6
2.30
43.1
MovieChat [7]
CVPR
2024
2.27
37.8
2.05
35.6
2.97
$7.9
2.15
43.0
2.36
43.5
Vidco-ooline [11]
CVPR
2024
36.5
38.2
41.1
LongVA [20]
Arxiv
2024
2.48
47.1
2.74
45.4
2.98
$7.8
2.22
42.4
2.60
48.1
LLaVA-Hound [29]
Arxiv
2024
2.69
48.7
2.56
43.7
3.07
56.8
2.42
42.7
2.68
47.9
Fr [8]
Arxiv
2024
2.48
46.7
2.32
41.7
3.02
58.1
2.16
38.3
2.49
46.2
Flash VStream [10]
Arxiv
2024
2.02
37.3
2.06
36.1
2.91
56.1
2.08
39.8
2,26
42.3
SIREANCEAT
2.78
50.1
In the MSVD and MSRVTT benchmarks, our short-term memory module allows the model to capture more specific visual details, leading to accuracies of 58.7% and 43.4%, respectively.
With the integration of long-term memory module, our model enhances the longer video perfor- mance, surpassing the previous best streaming method Flash- VStream [10] for 12.8% and the best offline method VAL-VA-ooun [29] for 1.4% % in ActivityNet [14] benchmark. In NExT-QA [15], our method can further improve the foundation model Long VA [20] by 5.1 % in accuracy.
Although the base model LongVA VA [20] has achieved the best average accuracy in offline benchmarks, method further improves it by 2.5 %, proving the effectiveness of the memory module.
4.3
CASE STUDY
In Fig. 6, we illustrate the reasoning process of STREAMCHAT with = 2 to simplify the observation of internal mechanisms. The scenario involves a user asking STREAMCHAT identify a tool meeting specific requirements and to describe its environment. The memory structure consists of a dialogue memory, Ma. with two historical entries, and layered memory, Mf, with two levels. The memory tree visualization shows that the system initially searches for key information at level 1. It computes cosine similarity between the user's query Q and two memory units, Summary (1) and (2), obtaining scores of 0.3993 and 0.4751, respectively. Based on these results, STREAMCHAT selects the path from the second node (w1) due to its higher similarity score and continues along this path. Subsequently. the system aggregates value {u,) ) ) from M, into retrieved tokens that then incorporated into the reasoning process. Additionally, a high similarity score of 0.6983 between Q: and the first historical conversation helps provide context, enhancing the depth and relevance of the response.
4.4 ABLATION STUDY
Exploring Effects of Hierarchical Memory. We conduct ablation experiments using the Base model to assess the impact of different memory components on performance. As shown in Table 7, adding M, to the base model improved performance the CI task by 4. 4.1% without affecting other
8

---

Published as a conference paper at ICLR 2025
Search histories
Eor Owary of
dless
Secoed Billege
pluce
peeer culer Can
Ma
en plrdiees pood
Thought
strigs Thes actocia og ynaran
(0p
Wetei welee muth
Search in long memory free
Integration
the the
Figure 6: An inference example of Stream ($4.3). Given a question, our system retrieves the most related information in long memory tree and dialogue histories based on the highest cosine similarity.
Table 7: Analysis of hierarchical memory. This table shows the impact of various memory configurations.
OS
LM
SM
Ci
KG
SS
Average
Ma M,
Acc.
Acc.
Sco.
Scv.
Scn.
Acc.
X
X
2.54
41.6
2.55
45.5
2.93
52.5
3.30
60.1
4.44
89.9
3.79
72.6
3.27
60.3
41.9
45.7
52.5
64.2
4.44
3.78
3.32
/
x
2.58
43,3
2.62
46.6
3.09
55.7
3.31
00.7
4.39
88.1
3.68
69.8
3,28
60.7
X
2.85
49.5
2.78
51.7
2.96
53.5
3.32
61.1
4.42
88.4
3.65
69.4
3.33
62.2
x
2.91
50.4
2.88
53.0
3.10
56.0
3.55
63.4
4.36
87.6
3.58
68.7
3,39
63.1
/
V
2.93
50.5
2.87
52.9
3.15
56.1
3.82
67.6
4.37
87.9
3.56
68.8
3,42
63.8
tasks. Adding Mr improved the LM task performance by 6.2%, while the use of M,, boosts SM task performance by 3.2%. The results indicate that the model's performance in each subtask aligns with the inclusion of specific memory attributes. Additionally, we observe that different memory components can complement each other. When both long-term Mi and short-term memory M, are applied simultaneously, the average accuracy increases by 0.9%.
Tradeoffss in Speed and Threshold Settings. The threshold of the Kancae Optical Flow algorithm significantly influences video processing speeds. As illustrated in Fig. 7 (a), increasing the threshold initially accelerates the processing speed. However, this increase saturates when l reaches 0.55, stabilizing at 32 FPS. Importantly, higher processing speeds are discouraged due to their detrimental impact model performance (64.0%->60. pronounced changes in frame differences and loss of original data, thereby limiting the model's ability effectively utilize the full spectrum of video information.
Elevating thresholds leads to more
Design of Long Memory Tree. The chunk length (L), group size (g), and clustering goal (C) significantly impact the effectiveness of the memory tree (M)) In Fig. 7 (b-d), we evaluate how these factors influence online video understanding tasks, using the Base model with t=0.35.
As shown in Fig. 7 (b), increasing L form 15 30 leads to better performance However, further increasing L to 40 results in a slight decline (64.0%-<6.. 1...%.... and substantially increases latency (0.84s 1.26s), due to the computational demands of the clustering algorithm.
Increasing g from 2 to 12, which represents the less compression of visual information and increases input sequence length C' x L,, enhances performance (62.0%-63..9%% as greater diversity in the knowledge at each node of the long memory tree M, is achieved. However, it intensifies the load on the retrieval, leading an increase in RDP (0.76s-+1) -.00s), illustrated in Fig. 7 (c).
The clustering goal (C) primarily influences the number of tokens (1;) stored in the M,. Fig. 7 (d) shows that increasing the dimension of v; (3-10) enhances model performance (59.4% +64.0% by enriching the stored knowled ledge,, which also exacerbates VRAM limitations (20~556 GB).
1.4)
64.5
64D
183
61.0
64.0
50
182
112
433
60.5
63.0
0.3
53.0
30
I82
6m0
62.0
58.0
Auc
Speod
52
Arc
RDP
640
61.
Acc
R/P
Menury
015020015 55 56
46
20
(a)L-25 9- 10 C-5
(b) 0- 10
G-5
(c)
f-335
L-25
C+5
(d) f 0035
L-15 g- 10
Figure 7: Analysis of memory parameters. (a) The influence hetween speed and threshold; Impact of (b) chunk
length and (c) group size on performance and latency; (d) Effect of clustering goal on performance and VRAM.

---

Published as a conference paper at ICLR 2025
5 RELATED WORK
Multi-modal Language Models (MLMs). Recent developments of large language models [3, 34-
38] and multi-modal alignment techniques have significantly advanced ML.ss capability [39-44].
The LLaVA series [45, 46] utilizes straightforward mapping layers and visual instruction tuning to
broaden image understanding tasks to video data. Challenges in video processing primarily involve
efficiently compressing video within limited contextual windows. Innovations like ChatUniVi's [5]
use of a K-NN clustering algorithm dynamically compress visual tokens, while LLaMA-VID [2]
reduces single images to two tokens via cross-attentien and MovieChat [7] leverages long and
short-term memory frameworks for extensive data handling. Despite these advances, the transition to effective real-time streaming video understanding in practical applications remains insufficiently addressed. Our research introduces robust solution designed meet the real-time demands of
online video understanding. aiming to fill this critical gap.
Streaming Video Understanding, Streaming video understanding demands real-time responses from models to user queries, even as video durations potentially extend indefinitely. This is particularly challenging for traditional benchmarks like action recognition [47]. multi-round video dialogue [48], and first-person question answering [13] which rely on uniform frame sampling. In response to these limitations, there is a growing shift towards online models that process only current and past video frames to formulate responses [10, 11]. Despite these advancements, these models often struggle with slow processing speeds and inadequate generalization capabilities, underlining a critical need for further exploration and enhancement in this field.
Retrieval-Augmennter Generation (RAG). RAG combines information retrieval and text generation to produce more precise and informative responses by incorporating external knowledge into language models [49-59]. This technique has become increasingly popular for addressing knowledge retention and real-time information access challenges. MemoryBank [60] enhances interaction by storing real-time conversations and leveraging similarity search to retrieve contextually relevant information, enriching the depth and coherence of dialogue. This approach significantly improves model's ability to maintain continuity in conversations, particularly in long or multi-turn interactions where maintaining context is crucial. Inspired by RAG's efficiency, system that integrates and updates textual and visual data in real time. Using a RAG-inst -AGGiisppred retrieval mechanism, this system efficiently accesses the most relevant information from our memory bank, enabling the multi-modal language model to deliver precise, query-specific responses for enhanced video language understanding.
introduce a multi-moda memory
6 CONCLUSION
In this work, introduce STREAMBENCH, comprehensive benchmark specifically crafted to assess streaming video understanding, covering a broader range of video lengths and types with six question formats to simulate real-world human-robot interactions. This broader scope enhances our ability to evaluate model performance in complex and dynamic scenarios. Alongside, we present STREAMCHAT, training-free method designed for efficient streaming video understanding, which treats video frames as compressible and storable units and manages them through a hierarchical memory structure. With advanced system scheduling, STREAMCHAT achieves real-time processing speeds and reduced interaction latency, demonstrating robust performance across both online and offline settings our extensive experiments.
Limitations and Future Works. Our current retrieval algorithm relies on basic matching techniques, occasionally leading to incorrect responses. Enhancing this with more fine-grained retrieval mecha- nisms is essential next step. Additionally, the VRAM constraints of our tree-structured storage could limit scalability as video duration and complexity further grow. Investigating more efficient or adaptive compression techniques will address these limitations. Moreover, achieve lower latency, we plan to explore closer hardware integration and the potential adoption of festing multi-modal distributed systems to accommodate larger model parameters and increased user demands.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work was supported in part by the National Natural Science Foundation of China under Grants 62406053, 62441231 and 62206039, and in part by the Fundamental Research Funds for the Central Universities under Grants DUT24YG119 and DUT24RC(3)025.
REFERENCES
[I] OpenAL. Chatgpt: Optimizing language models for dialogue. https:/ https://openai.... blog/ chatgpt. 2022. Accessed on: November 30, 2022.
[2] Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. arXiv preprint arXiv:231 1.t::22111.77443 2023.
[3] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
[4] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:231 2023.
[5] Peng Jin, Ryuichi Takanobu, Wancai Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13700-13710, 2024.
[6] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306. 054244 2023.
[7] Enxin Song, Wenhao Chai, Guanhong Wang. Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye. Yanting Zhang, et al. Moviechat: From dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18221-182333 2024.
[8] Wenhao Wu. Freeva: Offline mllm training-free video assistant. arXiv preprint AXXi:2445.007996 2024.
[9] Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.1663 16994, 2024.
[10] Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. arXiv preprint arXiv:2406.08085. 2024.
[11] Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao, Jia-Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online: Online video large language model for streaming video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18407-18418, 2024.
[12] Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5288-5296, 2016.
[13] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding. Advances in Neural Information Processing Systems, 36, 2024.
[14] Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127-9134, 2019.
[15] Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of question- answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9777-9786, 2021.
11

---

Published as a conference paper at ICLR 2025
[16] Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pages 1645-1653, 2017.
2017
[17] Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of mudti-modal Ilms in video analysis. arXiv preprint arXiv:2405.21075 2024.
[18) Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-task long video understanding. arXiv preprint arXiv:2406.0m 0446-4 2024.
[19] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijaynnarasimaan Youtube-8m: A large-scale video classification 2016.
benchmark. arXiv preprint arXiv:/609.0m
[20] Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang. Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprint arXiv:2406. 16852, 2024.
[21] Bruce D Lucas and Takeo Kanade. An iterative image registration technique with an application to stereo vision. In IJCA/'81: 7th international joint conference on Artificial intelligence, volume 2. pages 674-679, 1981.
[22] Richard C Atkinson. A proposed system and its control processes. The Psychology of Learning and Motivation, 2, 1968.
[23] Hermann Ebbinghaus. Memory: A contribution to experimental psychology. Annals of neurosciences, 20(4):155, 2013.
[24] Wenhui Wang. Furu Wei, Li Dong, Hangbo Bao, Nan Yang. and Ming Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. Advances in Neural Information Processing Systems, 33:5776-5788, 2020.
[25] Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, 7(3):535-544 547, 2019.
[26] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. 8763. PMLR, 2021.
[27] Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21226,, 2024.
[28] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[29] Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward. arXiv preprint arXiv:2404.01258. 2024.
[30] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Miniepm-v: A gpt-4v level milm on your phone. arXiv preprint a4Xiv:2408. 01 01800, 2024.
[31] Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21689, 2024.
[32] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistir tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24185-24198, 2024.
[33] Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, al. Internlm-xcomppoer--d A versatile
12

---

Published as a conference paper at ICLR 2025
large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320, 2024.
[34] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 18:110771,,,, 2020.
[35] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint mrXiv:2205.00006 2022.
[36]
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling instruction-finetune language models. Journal of Machine Learming Research, 25(70):1-53, 2024.
[37] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprini arXiv:2305. 10403, 2023.
[38] R OpenAl. Gpt-4 technical report. arxiv 2303.08774. View in Article, 2(5), 2023.
[39]
Wenguan Wang. Yi Yang. and Yunhe Pan. Visual knowledge in the big model era: Retrospect and prospect. Frontiers of Information Technology & Electronic Engineering, 2024. In press.
[40] Zongxin Yang. Guikun Chen, Xiaodi Li, Wenguan Wang, and Yi Yang. Doraemongp:: Toward understanding dynamic scenes with large language models (exemplified as a video agent). In Forty-first International Conference on Machine Learning, 2024.
[41] Wei Li, Hehe Fan, Yongkang Wong, Mohan Kankanhalli, and Yi Yang. Topa: Extend large language models for video understanding via text-only pre-alignment. In NeurlPS, 2024.
[42] Sitong Gong, Yunzhi Zhuge, Lu Zhang, Zongxin Yang, Pingping Zhang, and Huchuan Lu. The devil is temporal token: High quality video reasoning segmentation. arXiv preprint arXiv:2501.0844 2025.
[43] Haomiao Xiong, Yunzhi Zhuge, Jiawen Zhu, Lu Zhang, and Huchuan Lu. 3ur-llm: An end-to-end multimodal large language model for 3d scene understanding. arXiv preprint 250i::2501.08//1 2025.
[44] Junwen He, Yifan Wang, Lijun Wang, Huchuan Lu, Jun- Yan He, Jin-Peng Lan, Bin Luo, and Xuansong Xie. Multi-modal instruction tuned Ilms with fine-grained visual perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13980-13990, 2024.
[45] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neurul information processing systems, 36, 2024.
[46] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction wning. Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pages 26296-26300 26306, 2024.
[47] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308, 2017.
[48] Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, et al. Audio visual scene-aware dialog. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1558-7567) 2019.
[49] Penghao Zhao, Hailin Zhang, Qinhan Yu, Zhengren Wang, Yunteng Geng, Fangcheng Fu, Ling Yang. Wentao Zhang, and Bin Cui. Retrieval-augmented generation for ai-generated content: A survey. arXiv preprint arXiv:2402. 19473, 2024.
[50] Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao and Nan Duan. Query rewriting for reurreeal-augmenteed large language models. arXiv preprint arXiv:2305.14222 2023.
[51] Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators. arXiv preprint arXiv:2209, 10063, 2022.
13

---

Published as a conference paper at ICLR 2025
[52] Zhihong Shao, Yeyun Gong. Yelong Shen, Minlie Huang, Nan Duan, and Weizhu Chen. Enhancing retrieval -augeettee large language models with iterative retrieval- gennrratonn synergy. arXiv preprint 2.N5::2305.1299 2023.
[53] Xintao Wang, Qianwen Yang, Yongting Qiu, Jiaqing Liang, Qianyu He, Zhouhong Gu, Yanghua Xiao, and Wei Wang. Knowledgpt: Enhancing large language models with retrieval and storage access on knowledge bases. arXiv preprint arXiv: 11761, 2023.
[54] Zhuyun Dai, Vincent Y Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith Hall, and Ming- Wei Chang. Promptagator: Few-shot dense retrieval from examples. arXiv preprint arXiv:2209. 1755, 2022.
[55] Zhiqing Sun, Xuezhi Wang. Yi Tay, Yiming Yang, and Denny Zhou. Recitation-augmentt language models. arXiv preprint arXiv:2210.01296 2022.
[56] Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint (rXiv:2310.113:: 1511, 2023.
[57] Xi Victoria Lin, Xilun Chen, Mingda Chen, Weijia Shi, Maria Lomeli, Rich James, Pedro Rodriguez, Jacob Kahn, Gergely Szilvasy, Mike Lewis, et al. Ra-dit: Retrieval-augmented dual instruction tuning. arXiv preprint 230.0:2310.01352, 2023.
[58] Oded Ovadia, Menachem Brief, Moshik Mishaeli, and Oren Elisha. Fine-tuning or retrieval? comparing knowledge injection in llms. arXiv preprint arXiv:2312.05594, 2023,
[59] Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma. Xinran Zhao, Dong Yu, and Hongming Zhang. Dense retrieval: What retrieval granularity should we use? arXiv preprint arXivv2232.00m 066448, 2023.
[60] Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19724-19731, 2024.
14

---

Published as a conference paper at ICLR 2025
SUMMARY OF THE APPENDIX
This appendix contains additional details for the ICLR 2025 submission, titled 'Streaming Video Understanding and Multi-round Interaction with Memory-enhanced Knowledge', which is organized as follows:
SA presents the dataset collection pipeline.
$B visualizes more cases our benchmark.
$C offers the details about retrieval algorithm.
$D introduces the details and calculation method of metrics.
S shows failure cases and analysis.
SF discusses model selection and deployment strategies.
$G outlines our plans for benchmark expansion.
A DATA PIPELINE
Step Classfication
Step Human Judge
video time line
video
video 2
video
fromz
Please help contgorize the video info ene the follewing entegeries (ackeng Eoonstruction CRoom- b.Ser dering. Others).
Video 1: Geokingg Vides Garden
VidesALLIA
Humon
multi dej E
save video
dufubooee
Step 3: Ansotation and Check
extrrac video
dotcbuse
Check
reWeR
EXTRISTT
detsbaate
Annotater
Amactotione
Finel
Figure 8: The date preparation pipeline utilized in STREAMBENCH.
Fig. presents our video collection pipeline. It consists of parts: (1) Classification; (2) Human judge; and (3) Annotation check. First, MLLM is utilized to complete the video classification based on our requirements. The following prompt used during the first data filtering step:
Based on the observed video information categorize the video into one of the predefined categories listed in Respond exclusively in the format of a Python dictionary string with the keys 'pred' and 'score'. The 'pred' key should contain the uppercase STRING of the chosen category. Refrain from providing any additional text or explanatory output. Your response should strictly follow this example: ('pred' 'A")""
(All_Class) is the options formation. When dealing with different datasets, we need to change the options. For example, when dealing with Youtube-8M [19], is (A: Drama, B: Action, C: Cartcon, D: Romance, E: Sci-fi. F: Others) and (A: Cooking, B: Construction, C: Room-Tous dataset. We save the output worth noting that we also used the original category information in the YouTube data. The above classification process is used primarily for secondary classification of MovieClips data. For EgoSchema, we need to classify all original videos as they don't I contain category annotations.
D: Gardening, E: Others) } for EgoSchema [13] JSON file and then find categories from the file as needed. It
B MORE VISUALIZATIONS
In Fig. 9-12, we visualize several STREAMCHAT cases applied to different types of videos. Specifically, Fig. 9 illustrates egocentric video annotation, where the system engages interactive questioning
15

---

Published as a conference paper at ICLR 2025
based on the visual information captured from first-person perspective. In this figure, various types of annotation questions are showcased, including object identification, memory recall, and knowledge- based interactions. Each example contains six distinct questions, with the Simple Start (SS) question placed first, while the sequence of the remaining five questions varies randomly throughout the video. The questions explore aspects such as short-term memory, long-term memory, object search, and conversational interaction, allowing for a broad range of analysis. To further enhance clarity, specific frames are highlighted along with the locations of key objects referenced in the questions, enabling better understanding of how the system interacts with the visual context at different moments.
video
13s
20
35
42
50
Simple
Based what you you cooking my put? Short-lerm Ac saz dust you are casking somathng simiarr samsgges the pol.
A: Yes, Stere ss scall kaile with blee handle en the table.
Where did place the cup that just filld with ouffee gronds?
Long-term Memory
@ What the Ingredients sassage?
typically
Based
tost container stniales
S8s
Seasonligge Bindors;
Additionl Ingredients time: 12h
136
Tiss
1ITs
Tats
Q Based the comrent, what you thikk Converational Interaction
the main lngrediear I'n cooking this time?
believe the man ingredient you are cooking
sousags we've dissseal sal you preparing something
my glass Objrett
Kid
A/ You check to the slak
time: 153
Egocentric Video
Figure Visualization of egocentric video analysis.
Video time
( What is the man wearing black how and not weaning glasss boiddng in his band? A: The man holde glase bottie
Simple
Where bottle
Object Search
Nack A: it the tablle next the mas.
glown??
time:
Ginc: 90y
Bemm
Vitaminns
Knowledge Basa
handuct wearing yellow vest? telophene
Menory
A: Red wine comtano anall acsounts o'vitarie Bi, B12, and folare
Gnec 1248
136s
Idia
190s
155
194s
21de
Q: Where the aless with red liquid the the perso weering yellow ves beld?
Shart-tem) Memory time: 153
O.Arc dheie aay toels con be used for communication
Interaction
Video
Aheh glos many glase buttles
Figure 10: Visualization of web video analysis.
16

---

Published as a conference paper at ICLR 2025
Can yourtal whene planic water buttle It's nest to hammer on top of fron
Objeat Scarch
These
Bone: M8g
101s
1066
1156
118p
1955
person
smoking plant! Knowledge
harmer??
Menuey
At sneting plast, lanmer is esiental for takk Based each furning shoping flatioing meial
striking
with
Bie 120s
thnc: 2005
< conn quio remember, what osler was the cloting Lang term the man with slack glaese and brown buder who wearing gloves and stilking in tro aod with hamane??
Memery
shirt hat A: Ie is using the hammer alter the end of Iron red.
person
Conerasiionn)
Interaction
A: His dloting was blok
dine: 3004
the 381s
Working Video
Figure 11: Visualization of working vidco analysis.
v1000 Bine line
person
shirt?
Factial
Gine: da
43
44
Q Which ancken cisittations were amang the first to use borses transpartatins
Koswiege
Where bo pair plreed Object Soarch
A: Severnl ansicat civiliations was adopter heoes cumparttiio.
A: Thore is's of Nack glawes heing wom by the man standing close to the wall.
incliding the Indo-aarmuan peoples, the Sumerban, the Pgyplians, and the Illmee.
dmc 51s
Gine: 98s
10%
11%
134s
18
you soc,
groued
Momery
this appears
Convenstional
Intersetion
A: No, Loun't lone any tible lops.
tine: 120g
time: 185g
Movie Video
Figure 12: Visualization of movie video analysis.
C RETRIEVAL ALGORITHM
Inspired by the retrieval argumentation system [49], our approach enhances the model's capability to address complex queries by retrieving the most relevant information from long-term memory for contextual support. As outlined in Algorithm 1, compute the similarity, Sim, between the user's request, Q. and entries T,, in the memory. This process identifies the optimal path for accessing the most pertinent stored knowledge, Thest and Cbest- Leveraging efficiently focus on the highest-similaee nodes at each layer, minimizing the computational load by avoiding exhaustive sub-node calculations. The selected knowledge, Thest and Cbests is then
likelike storage structure,
17

---

Published as a conference paper at ICLR 2025
integrated with Q in mixed prompt format to as the final input for the multi modal language model, facilitating accurate response generation.
Algorithm Knowledge Retrieval from Long- Term Memory
1: Input: User request Q
2: Output: Best matching knowledge T and C'
3: Initialize similarity Simmaz <- -00
4: Initialize best path knowledge Test Cbest f- 0, 0
5: for each node in the tree structure do
6: Compute similarity Sim Tn) where Tù is the caption node
7: if' Sim T.) > Simmaz then
8:
Simmaz (2, Thest: Cbest <- Tny Cn
9:
> Update best match knowledge
end if
11: if' node has children then
12:
Continue to next level
13: end if
14:
end for
15: Reconstruct input for MLLM using Thett, Cbest and Q
16:
return Thest, Cbest
D DETAILS OF METRICS
We design the following metries to measure the model's ability stream video understanding:
(1) Score and Accuracy: To assess the semantic correctness of single-ture dialogue, using language models a mainstream approach [12, 14, 16-18). We also use this as a key metric in our benchmark. In our test benchmark, we use the open-source language model LLaMA-3 8B [3] Instruct version as our scoring model f. Here the prompt that we used during scoring:
Prompt = [ ("role": "systen" "content": "You intelligent chatbot designed
for evaluating the correctness of generative outputs for question-answer pairs. Your task is compare the predicted answer with the correct answer and determine if they match meaningfully. Here's 's how you can accomplish the task:
INSTRUCTIONS:
Focus the meaningful match between the predicted answer and the correct answer. Consider synonyms paraphrases valid matches.
Evaluate the correctness of the prediction compared
answer. ")
("zole": "system", "content" ; "Please evaluate the following video-based question- answer pair: Question: question; Correct Answer: answer: Predicted Answer: prediction Provide your evaluation only yes/no and score where the score is integer value between 0 and 5, with indicating the highest meaningful match.
Please generate the response in form of Python dictionary string with keys 'llama pred' and 'score', where the value of 'llama pred" is a string of 'yes" or 'no" and the value of 'score' is in INTEGER, not STRING. DO NOT PROVIDE ANY OTHER OUTPUT TEXT OR EXPLANATION. Only provide the Python dictionary string. For example, your response should look like this: ('llama pred': YOUR JUDGE, 'score': YOUR SCORE.)" }]
Table 8: Prompt given to the LLaMA-3 model for evaluation.
We organize the question Q, reference answer Rt and model's response M into the Tab. 8 formation and send scoring model, which then provides score the range of 0-5 and evaluates whether the model's 's response is semantically correct:
= SQQ, R, M), Acc
N Σ I(S, T)
A higher score S, and Ace indicates that the answer is closer to the reference answer.
(2) Coherence: Given that a single video may involve multiple rounds of dialogue, we need to evaluate the model's ability to provide coherent experience across different rounds. We introduce
18

---

Published as a conference paper at ICLR 2025
the coherence metric, which calculates the absolute value of the difference between the semantic scores St of different dialogues within a single scenario. The average of all these differences is used as the coherence metric. The calculation formula is follows:
1 N-1
C N Σ S, - S
where C' is the coherence score, .V is the total number of dialogue turns in the scenario, S, represents the semantic score of the i-th dialogue turn and |-] is the absolute difference between the semantic scores consecutive dialogue turns. It evident that a smaller C' indicates that the model provides a better coherence experience for the user.
(3) Request Processing Delay:
For online scenarios, system latency consists of two parts: 1. Request processing delay; 2. Generation delay. The generation delay is mainly influenced by factors such as context length, language model parameters, and deployment methods, and can be adjusted through various methods. In this benchmark, we primarily assess (I) request processing delay, which calculated as the time from when the user completes the request input to when the model starts generating the response. The calculation formula as follows:
RPD = Tuan
Tinput
(8)
E FAILURE CASE AND ANALYZE
We present some failure cases that occurred during testing and explain why they occurred. Most of these cases come from object search, long-term memory, short-term memory and conversational interaction tasks. The problems that occurred are mainly grouped into four types:
Temporal Fine-grained: In the object search task, our method still struggles to identify key information when the queried objects or events appear too briefly or sporadically. For instance, as demonstrated in case (1), the user's question pertains to candle. However, due to the infrequent appearance of the candle and its small size, the model fails to provide an accurate response.
Spatial Fine-grained: Whether in object search or short-term memory task, our method faces limitations when the user's target is too small blends into the background, even if the object appears multiple times in the video. For example, in short-term memory task case (2) and object search task case (3), the target objects (porcelain bowl and red cup) foreground, making it difficult for the model to accurately detect and locate them. We will continue improve our method enhance the perception of small objects.
small relative the
Target Movement:
During the reasoning process, we observed that in some cases, even when the model correctly identified the target, its interpretation of the target's s actions and relationships with surrounding objects was still inaccurate. For instance, in the long-term memory task case (4) the model failed to recognize the action and position association between the "person" and the "box," leading to an incorrect response.
Context Induction: In the conversational interaction task, the model's performance is influenced by the accuracy of its responses to previous related questions. For instance, in case 5, the model retrieved information from the dialogue history, but when that historical information was incorrect, it became challenging for the model provide the correct answer.
Information Loss: According to the experimental results in Tab. although our method shows balanced performance in various tasks of StreamChat, our hierarchical memory storage still has the potential risk of losing information. Since our existing method too dependent on the accuracy of the retrieval algorithm, we will continue to update our method to minimize information loss.
F MODEL SELECTION AND DEPLOYMENT
Our research indicates that suitable model should possess the following key attributes:
Long-Form Video Understanding:
Effective processing of long videos is crucial. While we utilize K-Means for feature compression, the information retrieved by memory mechanism remains relatively long, requiring a model capable of handling extended sequences.
Robustness to Prompt Variations: For accurate and coherent multi-turn conversations, the model must be robust changes in prompt wording. This robustness is essential prevent inconsistencies
19

---

Published as a conference paper at ICLR 2025
The image informaion does
table people sitting
Temporal Fine-grained
white candles!
MELM
contain visible white caniles
Cin you tell mo where you porcelain howls?
No, did not see any porcelain bowil the provited informationn
Yes, there are some porcelain bowls on the countertop. the right of the stove
Spatial Fine-grained
Where the painting Sools just
The paintng tools and the paint untaine ane likely
On bench from you. Jooking
puiis cortiine??
MELM
desigsated storage wurkspace, sach dediconod paisting station, where they would be kept after
person wearing striped
bos containe yellow, greee, red cherry- sized objects
persin
laside hand
MILMM
possible accurately purpose
shout
Target Movement
striped cluthes touched
Based the image provided, there is ns voibll red water cup with The image shows classroom setting with Individunis engaged in an educational activity. and the focus on lorge screen displaying lest or document. If looking foc specific object withis this eavironment, would not present the image shown,
There is water with nod ld the table next to wearing rod shari-lteved shiri, adjucent to slack servall books.
Where
water with
MILMM
(5)
Are there any cups duis video
No,
There red
red vides)
Context Induction
Figure 13: Visualization of failure
or ballucinations in the model's output when prompts are adjusted to incorporate information from the memory mechanism.
By integrating VA with proposed system, successfully extend its capabilities compass streaming video processing and multi-turn conversations while preserving these critical characteristics. As introduced in $4, we utilize GPUS to complete the deployment of our method. The main reason is that during system scheduling, we need to utilize tensor parallelism to distribute the computational load for efficient execution. Specifically, the (i) selective frame stacking thread and (iii) context summarization thread are running on GUPI while (ii) memory formation thread is running on GPU2. Thereffeee,,,, the compressed video tensors need to transmitted between different GPUs to ensure the stable operation of the system.
G EXPANSION PLAN OF STREAMBENCH
Video scale: We are trying to expand the number of videos contained in the StreamBench to reach a higher standard. We are working on expanding the number of videos to thousands while maintaining the diversity of video length and types.
Annotation scale: We continuing promote the development of high-quality annotations. Based on your suggestions, we will use manual annotation methods to expand the annotation of existing benchmarks to the order of ten thousand levels and also use manual inspection filter out toxic labels and erroncous information.
Diverse tasks: Given that the current benchmark only has single task type, we are continuing to expand the types of tasks included in the benchmark, including but not limited to multiple-choice questions, video captioning, and video grounding and etc.
20


Reviews:
[{"review_id": "8dDkg17t5R", "reviewer": "Reviewer_Lmv2", "summary": "This paper introduces StreamChat, a training-free framework for streaming video reasoning and conversational interaction, and StreamBench, a comprehensive benchmark for evaluating streaming video understanding across various media and interactive scenarios. Extensive evaluations on StreamBench and other public benchmarks show that StreamChat significantly outperforms existing state-of-the-art models in accuracy and response times, highlighting its effectiveness for streaming video understanding.", "questions": "Please refer to the part of weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "Nice paper. This work is comprehensive, provides significant contributions, and is well-written.\n\nThe biggest contribution is StreamBench. Previously, there was no suitable benchmark for streaming video understanding. StreamBench is a high-quality addition.\n\nFurthermore, StreamChat is also very interesting, demonstrating that an agent-like system can also achieve state-of-the-art results in streaming video understanding.", "weaknesses": "Overall, the paper is satisfactory, but I have some concerns regarding StreamChat:\n\n- What motivated the selection of LongVA as a foundation? What are the essential capabilities required for other Video-LLMs to serve as the base model?\n\n- Would fine-tuning LongVA with some streaming vision-language data be better than the current system?\n\n- Is it feasible to integrate the proposed hierarchical memory storage into a trainable framework?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YFAfX5joup", "reviewer": "Reviewer_9hDR", "summary": "This paper introduces a method for online interactions with videos through an LLM, and a benchmark suitable for evaluating streaming-style scenarios where the dialogue happens in real time with the video.\n\nContributions:\n- Video-based LLM which processes high framerate video (typically LLMs need to subsample severely the video due to computational constraints). This method uses optical flow magnitude to assess whether to process any particular frame (w.r.t. previous), therefore saving considerable amount of computation needed. The LLM also has a \"short/long\" term memory system.\n- New benchmark that should be useful since it's one of a kind, simulating multi-round interactions centered on the video content. 25h of content was used in the creation of this benchmarks, making it one of the larger ones available.", "questions": "- Do you plan to make this benchmark publicly available?\n- Do you plan to open source your method so that others may benefit from the engineering effort you've put into your method? (I am of course talking about academic researchers)\n- What the process of identifying the various categories STREAMBENCH encompasses? I am of course talking about the domains and classes of videos. \n- In the appendix, you very briefly mention that you're using a LLM to determine accuracy. The formula you present is unclear. (What's N, T?). What are you getting out of the LLaMA model? Also, do you use any particular prompt for the LLaMA model?\n-", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strong empirical results. This is a systems pape which is tied to a particular benchmark which was designed for it, so it's unsurprising that the proposed method works best. With that said, given the engineering effort put into it, it still is impressive, particularly because of the fact that this system could be used in a realtime scenario. \n\nI am personally excited by the new benchmark that is proposed in this paper perhaps even more so than the method, because it captures a whole class of interactions which were under-represented before.", "weaknesses": "I am kind of baffled by the language of \"our method can further improve our foundation model LongVA [20] by 5.1% in accuracy\". Does this imply that you're the authors of this method? (Please don't answer this question, but please be mindful of the language which might accidentally de-anonimize you).\n\nThe hierarchical design seems a bit ad-hoc. It would have been nice to spend a bit more time to motivate the various design choices. Currently, it sounds like the paper is in the \"this is what we did\" style, without spending too much into the motivation which led to the various choices made. I could easily imagine many other ways in which the hierarchical memory could have been implemented.\n\nThe abstract claims \"compression\" of video features, but reading the paper reveals that this is actually not at all the case. I find the abstract to be misleading and should be updated to reflect the reality of the methods involved.\n\nMinor comments:\n- I am having a hard time reading figures. For example, in Figure 1 I had to use a color picker to figure out which method was which line. In the interest of color blind people, (or partially color blind people like me), please use symbols (*, triangle, diamond, etc) or line styles in addition to colors to make the figures more accessible (either to color-challenged people like me, or when printed out in grayscale).\n- There's a typo \"Gine-grained\" in the appendix.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5bPHNxyLuX", "reviewer": "Reviewer_qLQa", "summary": "This paper presents a benchmark, named StreamBench, and as well a model, named StreamChat, for real-time streaming video LLM answering. The StreamBench comprehensively evaluates assorted capabilities of video LLMs, including spatial search, long & short memory recall, common knowledge reasoning, etc. The size of StreamBench is also larger than previously-proposed benchmarks. The authors also paid a lot of efforts in building the training-free StreamChat model, which bases on current video LLM LongVA and consists of novel techniques including hierarchical memory storage, optimized system scheduling. Evaluations on various benchmarks show that StreamChat enjoys better performance with lower latency compared to previous streaming models.", "questions": "See weaknesses. I am just at the borderline due to 1) the contribution of StreamBench and StreamChat, 2) the over-complicated design of StreamChat (being too engineering and RAG-like).", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The proposed benchmark, StreamBench, is a comprehensive benchmark testing various capabilities of real-time video LLMs. As far as I know, this is the first benchmark designed for online streaming video task evaluation.\n2. The proposed method, StreamChat, integrates frame selection, memory building and retrieval, and several optimizations into a video LLM in a training-free manner. Though designed complicatedly, the latency of StreamChat is still under control (under one second). Several benchmarks, including online StreamBench and offline video understanding tasks, show the effectiveness of the proposed StreamChat. The authors also put a detailed inference example, every components illustrated, in Fig 6. Also, the detailed ablations on threshold and memory parameters should be appreciated.", "weaknesses": "My major concerns comes from two sides:\n1. **The presentation**. The presentation of StreamBench is good but the part of StreamChat seems to need to be further improved. It is hard to tell what StreamBench is designed to do when understanding Fig 4 and Fig 5. Though there are both captions and texts written in the method section, it is still hard to relate text descriptions to components in the figures (especially for ones less familar with memory mechanism and RAG).\n2. **The complexity**. StreamChat seems too complicated: there are selective frame stacking based on motion vectors, vision-based short-term memory, caption-based long-term memory (also a clustering mechanism in it), dialogue memory (whose language encoder is different from video LLM), and FAISS indexing library. Although being tuning-free, all these components make this 'system' super heavy. The authors should justify every components used in building the memory system in StreamChat. Moreover, this looks more like a RAG system specifically designed for online videos. The authors should compare with other basic RAG methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N82zHf8ifZ", "reviewer": "Reviewer_Bg29", "summary": "In this work, the authors propose a training-free framework, StreamChat, for streaming video understanding. StreamChat uses a hierarchical memory system to efficiently process long video sequences for real-time, multi-turn dialogue. With parallel scheduling, it improves speed and reduces latency. \n\nThe authors also introduce a new benchmark, StreamBench, which tests video understanding across diverse scenarios, and results show StreamChat outperforms current models in accuracy and response time.", "questions": "Q1: Will the example in the prompt impact the models' prediction? For example, your example in the prompt is \"{'pred': 'A'}\". Does it increase the probability of predicting \"A\"?\n\nQ2: The StreamBench benchmark supports only QA tasks now. Are there any plans to extend it to include additional streaming video understanding tasks, such as dense streaming video captioning or temporal/spatiotemporal grounding?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 1, "strengths": "S1: The paper introduces a novel hierarchical memory mechanism that compresses video representations over long sequences, enabling efficient video feature retrieval in real-time multi-turn dialogue contexts.\n\nS2: I believe the introduction of StreamBench as a benchmark for streaming video understanding is a significant contribution that can fill a critical gap in the field. By offering a standardized evaluation framework, StreamBench enables more rigorous comparisons across models, driving advancements in streaming video understanding.\n\nS3: The proposed StreamChat achieves the best performance on StreamBench, surpassing other state-of-the-art methods, which is impressive.", "weaknesses": "W1: StreamChat was compared only with several open-source models. Including proprietary models like GPT-4o and Gemini-1.5 in the StreamBench evaluation would provide a more comprehensive comparison. \n\nW2: Human performance is absent in the comparison, which would help illustrate the gap between current models and human capabilities if included.\n\nW3: The authors compare their new benchmark only with older datasets or benchmarks, such as MSVD, MSRVTT, and ActivityNet. It would strengthen the evaluation to include comparisons with more recent benchmarks for video understanding, such as Seed-Bench [1], Video-Bench [2], MVBench [3], and LVBench [4].\n\nW4: The paper lacks an analysis of factors that impact performance on StreamBench, such as the input frame sequence length or the language model size used by the models.\n\n[1] Li, Bohao, et al. \"Seed-bench: Benchmarking multimodal llms with generative comprehension.\" arXiv preprint arXiv:2307.16125 (2023).\n\n[2] Ning, Munan, et al. \"Video-bench: A comprehensive benchmark and toolkit for evaluating video-based large language models.\" arXiv preprint arXiv:2311.16103 (2023).\n\n[3] Li, Kunchang, et al. \"Mvbench: A comprehensive multi-modal video understanding benchmark.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[4] Wang, Weihan, et al. \"LVBench: An Extreme Long Video Understanding Benchmark.\" arXiv preprint arXiv:2406.08035 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tKDwBz0OMj", "reviewer": "Reviewer_yVyo", "summary": "This paper studies streaming video understanding. The authors propose a new benchmark named StreamBench to evaluate streaming video understanding across diverse media types and interactive scenarios, including multi-turn interactions and complex reasoning tasks. They also propose StreamChat, a training-free framework for streaming video reasoning and conversational interaction, with a complex memory mechanism. Their method enhances processing speed and reduces latency, ensuring robust performance in real-world applications. Extensive evaluations on StreamBench and some public benchmarks demonstrate that StreamChat outperforms the selected\nbaselines.", "questions": "Please respond to Weaknesses.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed StreamBench may be the first benchmark for streaming video understanding.\n2. The proposed StreamChat outperforms the selected baselines on StreamBench.\n3. The processing speed of StreamChat significantly outperforms those of baselines.", "weaknesses": "1. The proposed dataset is too small, only 306 videos and 1.8K  question-answer pairs are collected. The current video benchmark typically has at least thousands of videos and tens of thousands of QA pairs. This version of the benchmark is not ready for release.\n2. State-of-the-art video LLMs are not included in the benchmark, such as MiniCPM-V 2.6 [1], InternLM-XComposer2.5 [2], VILA [3], and InternVL2 [4]. The effectiveness of the proposed method is unclear.\n3. The Selective Frame Stacking seems may ignore small objects in the video and only focus on the global frame feature.\n4. The proposed memory mechanism is very complicated and discards a large amount of information in the video.\n5. The authors test the processing speed of models on two NVIDIA Tesla A800 80GB GPUs, which is not a typical scenario of model development in real-world applications.\n\n\n[1] Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., ... & Sun, M. (2024). Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800.\n[2] Zhang, P., Dong, X., Zang, Y., Cao, Y., Qian, R., Chen, L., ... & Wang, J. (2024). Internlm-xcomposer-2.5: A versatile large vision language model supporting long-contextual input and output. arXiv preprint arXiv:2407.03320.\n[3] Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., & Han, S. (2024). Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 26689-26699).\n[4] Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., ... & Qiao, Y. (2024). How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)