Streaming Long Video Understanding with Large Language Models
Authors: Rui Qian
Xiaoyi Dong
Pan Zhang
Yuhang Zang
Shuangrui Ding
Dahua Lin
Jiaqi Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: LLM
Long Video Understanding
Memory-Propagated Streaming Encoding
Adaptive Memory Selection
Primary area: machine_vision
TL;DR: We propose a memory-propagated streaming encoding architecture with adaptive memory selection for long video understanding with LLM.

Abstract: This paper presents VideoStreaming, an advanced vision-language large model (VLLM) for video understanding, that capably understands arbitrary-length video with a constant number of video tokens streamingly encoded and adaptively selected.
The challenge of video understanding in the vision language area mainly lies in the significant computational burden caused by the great number of tokens extracted from long videos. Previous works rely on sparse sampling or frame compression to reduce tokens. However, such approaches either disregard temporal information in a long time span or sacrifice spatial details, resulting in flawed compression. 
To address these limitations, our VideoStreaming has two core designs: Memory-Propagated Streaming Encoding and Adaptive Memory Selection. The Memory-Propagated Streaming Encoding architecture segments long videos into short clips and sequentially encodes each clip with a propagated memory. In each iteration, we utilize the encoded results of the preceding clip as historical memory, which is integrated with the current clip to distill a condensed representation that encapsulates the video content up to the current timestamp. This method not only incorporates long-term temporal dynamics into the streaming encoding process but also yields a fixed-length memory as a global representation for arbitrarily long videos. After the encoding process, the Adaptive Memory Selection strategy selects a constant number of question-related memories from all the historical memories, and feeds them into the LLM to generate informative responses. The question-related selection reduces redundancy within the memories, enabling efficient and precise video understanding. Meanwhile, the disentangled video extraction and reasoning design allows the LLM to answer different questions about a video by directly selecting corresponding memories, without the need to encode the whole video for each question. Through extensive experiments, our model achieves superior performance and higher efficiency on long video benchmarks, showcasing precise temporal comprehension for detailed question answering.

Streaming Long Video Understanding with Large Language Models
Rui Qian Xiaoyi Dong" .2 Pan Zhang² Yuhang Zang² Shuangrui Ding'
Dahua Lin 1,2,3 Jiaqi Wang2t
1 The Chinese University of Hong Kong
Shanghai AI Laboratory
3 HKGAI under InnoHK
Abstract
This paper presents VideoStreaming. advanced vision- -aaiunne large model (VLLM) for video understanding, that capably understands arbitrary-length video with constant number of video tokens streamingly encoded and adaptively lected. The challenge of video understanding in the vision language area mainly lies in the significant computational burden caused by the great number of tokens extracted from long videos. Previous works rely on sparse sampling or frame compression to reduce tokens. However, such approaches either disregard temporal information in a long time span or sacrifice spatial details, resulting in flawed com- pression. To address these limitations, our VideoStreaming has two core designs: Memory--ropaggated Streaming Encoding and Adaptive Memory Selection. The Prooaay-Propaggaeed Streaming Encoding architecture segments long videos into short clips and sequentially encodes each clip with a propagated memory. In each iteration, we utilize the encoded results of the preceding clip as historical memory, which is integrated with the current clip to distill condensed repre- sentation that encapsulates the video content up the current timestamp. This method not only incorporates long-term temporal dynamics into the streaming encoding process but also yields a fixed-length memory as global representation for arbitrarily long videos. After the encoding process, the Adaptive Memory Selection strategy selects constant number of question-related memories from all the historical memories, and feeds them into the LLM to generate informative responses. The question-relatee related selection reduces redundancy within the memories, enabling efficient and precise video understanding. Mcanwhile, the disentangled video extraction and reasoning design allows the LLM to answer different questions about video by directly selecting corresponding memories, without the need code the whole video for each question. Through extensive experiments, our model achieves superior performance and higher efficiency on long video benchmarks, showcasing precise temporal comprehension for detailed question answering.
Introduction
The evolution of Large Language Models (LLMs) has significantly advanced artificial intelligence, encompassing text generation and reasoning complex language environments 74 14
16
69 22 [70] Later, the community extends LLMs to multi-modal domains, demonstrating promising results captioning and guestion-answering answering tasks that integrate diverse visual signals 46 40
B 59].
Yet, within the domain of video understanding, long video sequences pose a formidable challenge.
Work done during an internship in Shanghai AT Laboratory. 1 Corresponning Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Incorporating such long visual contents into LLMs requires a substantial number of tokens, which not only amplifies computational demands but also risks early contextual information loss 49
Among the recent works on general video understanding with LLMS 48
50
43
66
62
prevalent strategy is using sparse temporal sampling [44 [6 or spatio-temporal pooling 150
48 to
reduce tokens. Unfortunately, this paradigm explicitly loses substantial information in the long time
span. To address this limitation, [43 42 88 develop frame-wise compression, with LLaMA-VID 443
as a typical example. It compresses each frame into only two tokens but overlooks the inter-frame temporal dynamics which are vital in compressing temporal redundancy within videos. Besides, its -eeesionn-eeeddentt compression pipeline limits the ability to produce general representation that can handle diverse instructions. Another line of works employ memory banks [77 7 to store history information [66] [2]. Whereas, these methods rely on explicit timestamps to recall the historical details, limiting the ability to generate comprehensive responses without specific time indicators.
In this work, we propose VideoStreaming, a novel Hemoor-Propagatee Streaming Encoding architec- ture with Adaptive Memory Selection to sequentially encode a long video into condensed memories and generate responses referring to relevant timestamps. The core idea behind the memory- propaaa-d streaming encoding to preserve representative spatial cues and temporal dynamics while reducing temporal redundancy in videos. To achieve this goal, we segment the long video into multiple short clips and sequentially encode each clip. When encoding each clip, we first refer to the encoded results of its preceding clip as historical memory, then concatenate it with the current clip features and feed them into a small decoder-only language model 20].. Due to its autoregressive nature, the information of the sequence naturally accumulates to the last few tokens [8 34]. Consequently, we take these last few tokens as an updated memory that encapsulates the video information up to the current timestamp. Through this streaming encoding, we explicitly take long-term temporal relations into consideration and maintain a fixed-length memory to represent an arbitrarily long video.
However, this fixed-length memory inevitably loses detailed information, especially in early contexts. To address this problem, we store the historical memories of all clips and select constant number of subsets that are closely related to the question. To accomplish this, when streaming encoding each clip, we additionally append a summary token at the end of the sequence as clip indicator that summarizes the clip contents within one token. Then, given a specific question, we concatenate the condensed memory from the final iteration with the question and pass it through the same small language model used in streaming encoding. We take the final token as the question indicator and calculate its similarity with all historical clip indicators, the clip indicator with higher similarity means its corresponding memory is more related to the question. Finally, we feed the adaptively selected memories into the LLM for detailed question answering.
In practice, we realize our VideoStreaming with a carefully designed two-stage progressive training process and long-video video data construction strategy. In the first stage, we empower small language model with the single-clip encoding capability by a specialized prefix task. In the second stage, it serves as the streaming encoder and we jointly train with the LLM for long video understanding. Due to the lack of long video QA data, we manually constructed set of long video QA pairs in two ways. On the one hand, we concatenate short videos from existing datasets 77 8T into longer ones, where the original questions correspond to different segments. On the other hand, we curate a subset of Panda-70k 70M [ which includes captions for segmented clips as well as the original long videos, and use this to create multi-round long video QA pairs with explicit timestamps. These long video QA data not only optimize the responses from the LLM but also guide accurate memory selection.
In summary, our contributions are as follows: (1) We analyzed the challenge of long video under- standing in the vision language area, and pointed out that the problem of current methods lies in the inefficient video encoding. (2) In response to the challenges, we propose two efficient designs: Streaming Encoding and Adaptive Memory Selection, which result in our advanced video understanding model VideoStreaming. (3) The extensive experiments demonstrate that our model achieves precise temporal grounding with respect to specific questions, attains superior performance, and exhibits higher inference efficiency on long video benchmarks.
2 Related Work
Large Language Models (LLMMs) have revolutionized natural language processing. Farly works establish encoder-decoder models with masked language modeling [16] 61], while later decoder-only
2

---

models like GPT [60] showcase remarkable performance and scalability. Recent groundbreaking works, such as PaLM 14, LLaMA [70] and GPT-4 [56]. have pushed the boundaries by developing significantly larger models with billions of parameters. To harness the full potential of L.MMs, a series of works [55] 57 adopt supervised instruction tuning [74 to guide models towards generating more natural and contextually relevant responses. Inspired by the powerful reasoning capacities of LLMs, we explore using LLMs for challenging long video understanding.
Vision Language Models like CLIP 39
employ contrastive learning on image-text pairs to formulate 39
a unified embedding space [59
85] integrate image features into LLMs and achieve promising visual reasoning. To process more complex video data, [50] 48, 44
28
Later, [46
56 90
25
use sparse sampling or simple temporal pooling to obtain compact video tokens for LLMs. employ Q-Former [39 to project frame-wiss features into the textual space. To handle longer videos, 31
86
42
g8
75 utilize token merging 8 to reduce redundancy and alleviate computational burden. proposes an instruction-avee
LLaMA-VID 4
compression strategy to represent each frame with
only two tokens, but it overlooks the temporal relations in the compression step. [66 22 develop memory banks to accumulate information in long videos and excel in global video comprehension. However, these methods struggle with moment-specific questions without explicit time indicators. To address these limitations, we propose meeorry-rooaggate streaming encoding architecture with adaptive memory selection, which effectively reduces temporal redundancy and accurately selects relevant information for detailed question answering.
Long Video Understanding is a challenging task computer vision. The most prevalent strategy is to maintain memory bank to store history information in long videos [77 7 78
54
12]
71
21. Typically, MemDPC (2 formulates memory bank shared across the whole dataset as an external knowledge reference for future prediction, but it is not explicitly designed for long-term understanding. To facilitate long video content analysis and ensure computation efficiency, MC- ViT 77. MovieChat [66] and StreamingCaption [89] rely on handcrafted rules like clustering to compress the history into a finite-length memory and iteratively update it video streams. potential drawback is that the memory cannot be optimized through the extensive video caption data. It is potentially more promising to learn comprehensive memory consolidation in a data-driven manner. More recently. [32 27 87 use language
bridge for long-term video understanding. They first divide a long video into short clips. generate textual descriptions for each clip, and then employ an LLM aggregate the short captions for long video analysis. However, this architecture cannot be trained end-to-end, and the long video understanding quality depends on the short clip captions. In contrast, we employ a trainable small language model to iteratively encode short clips into compact memories, which can be jointly optimized with the subsequent LLM for long video understanding.
3 VideoStreaming
In this section, we introduce VideoStreaming, a streaming long video understanding framework with LLM. As illustrated in Fig. Tal given a long video input, VideoStreaming segments it into multiple short clips and iteratively encodes each clip into compact historical memory. To enhance the reasoning ability to specific questions, we design an adaptive memory selection strategy select a subset of relevant memories and feed them into an LLM to produce detailed responses.
3.1 Single Clip Encoding
To effectively distill the information within a sequence into a compact set of tokens, we take
inspiration from recent advanced decoder-only language models [2] 70 13 5 29 and employ comparatively small language model, Phi-2 [20]. for efficient encoding. Due to the causal attention and autoregressive nature, it is intuitive to utilize language model to aggregate the sequence information onto the last few tokens [38] 34], which naturally serve as a compact representation that provides a high-level summary of the input sequence.
Mathematically, given T-faame video clip, we first use a pre-trained CLIP ViT-L [59 to extract frame-wise features and concatenate every four spatially adjacent visual tokens along channel dimension to reduce the number of tokens by 75%. The resulting clip features are denoted as F E RTNNC xC, where N denotes the per-frame spatial token number, and C is the channel dimension. To produce the condensed representations, we initialize a set of summarization tokens S E PxC by adaptively pooling each frame into P tokens, where P << N. Intuitively, S can be regarded as
3

---

Assistant: man ratses nis left orm in misdle of video grab buby'y hand.
Condensed representation
Clip indicator
Large Language Model
Small Language Model
1X
H Histurical memory
Tokenizer
Projector
Streaming encoder
Streaming encoder
Streaming enteader
Streaming enooder
Tokenizer
contnte élstary of to ftui
Frame-wise wiie clip Summarization festlures
tokkens
F1
F2
M2-1
R
necondl, and clpp
Vision enooder
Vision encoder
Vision enceder
Tokanizer
Vision encoder
Pooling
fa-s
User: Why dloss the in the middle of the video?
aecondl.
Time promitt
Segmented clips a long video
I-th clip
(b) Details of the streaming encoder. Figure I: Fig. Ia shows overview of VideoStreaming, where we segment a long video into short clips and iteratively encode each clip into compact memories. Then, according to specific questions, we select constant number of subsets of relevant memories as input an LLM to produce responses. The / and X respectively denote selected and unselected memories. Fig. 1b illustrates the detailed process of each streaming encoding iteration. We encode current clip features with reference to specific timestamps and historical memory from the preceding clip into condensed representation.
(a) An overview of streaming video understanding framework.
coarse encapsulation of the given clip, making it well-suited serve the summarization tokens for consolidating the clip information. To this end, we concatenate F with S and feed them into the encoder g(-), which consists of MLP projector and language model Phi-2. We utilize the output of the last T x P tokens as the condensed representation of the given clip:
H g S)) TRXX
where denotes concatenation operation, D is the channel dimension of Phi-2.
To reinforce the visual consolidation ability, we design a prefix task to train the encoder on visual captioning and question question-answering answering tasks. In particular, guarantee that the clip information is distilled into the summarization tokens, we enforce the language model to generate the response only with reference to these few tokens. To achieve this goal, straightforward way is to modify the attention mask in each Transformer decoder layer. As depicted Fig. 2 we take a sequence covering TN clip feature tokens, TP summarization tokens, and TT text response tokens
example. Based on the standard causal attribute, the binary attention mask M is modified as shown in Figure 3
H 7PPx
Text response
Clip frorree tokens
TN
Small Language Model
Summarization
FE TCN
E TPCC
Text response
Test resporse toker
Prefix
Figure 2: Illustration of the prefix task format.
Figure 3: Modified attention mask M.
with the modified attention mask, the TT text tokens can only get video-related information from the TP summarization tokens to predict the next token. This encourages the summarization tokens to extract more video information from previous TN video clip tokens, ie. learns better video encoding.
3.2 Memory-Propagater Streaming Long Video Encoding
Till this point, we have obtained an encoder capable of distilling short video clips into condensed representations. The next step is to comprehensively consider the long-term temporal relations within the complete videos, leveraging the historical information from previous clips to facilitate the encoding of subsequent segments depicted in Fig. lb
To accomplish this objective, we divide a long video into K clips, each containing T frames, and propose memory -ronnoor--oopgatedd streaming encoding mechanism to iteratively encode each clip in

---

sequence. In each iteration, we employ the encoded results from the last iteration as historical memory and integrate them with current clip features produce an updated memory for subsequent encoding. Specifically, given the k-th clip, we denote the current clip features as F& E TRTNNC, the summarization tokens as Sk E IRT'XC and an additional global token as S₁ E I This global token, initialized by global average pooling on the clip features Fk, is expected to summarize clip indicator for memory selection in the next subsection.
the entire clip contents and serve To enrich the temporal contexts, we refer to the encoded representations from the previous clip H RTPxx D to provide historical information. Then jointly feed them into the streaming encoder produce the condensed representation H₂ IRTPx and the clip indicator H2 RIxD of the k-th clip:
- - F2 S Stl).
(2)
Note that for the first clip encoding, the historical memory is not used. Through this streaming encoding process, H2 not only encompasses the current clip information but encapsulates the overall video content up to the k-th clip. To this end, we manage to maintain a fixed length of memory to represent arbitrarily long videos.
Discussion. In this architecture, we use a language model for video encoding, which has the unique advantage that we can flexibly provide the encoder with diverse prompts to guide the encoding process. Hence, the summarization tokens capture not only the core content but also additional contextual information. Typically, the explicit timestamp is an important cue in videos [2] As shown in Fig. Tb] we incorporate a text prompt indicating the specific timestamps of each clip and historical memory to enhance temporal awareness. Besides, this prompt-based approach also allows the user to tailor the condensed output to better suit the needs of downstream tasks, going beyond a purely extractive summarization.
Another noteworthy point is that in the language model, the feature space of the final decoder layer is designed for the next token prediction, which may not perfectly align with the objective of producing condensed video representations. Considering that we modify the attention masks in each decoder layer to encourage information consolidation, this allows us to leverage the intermediate outputs from partial attention layers as the encoded results. Similar to the techniques in vision domain [82 46 17].
this strategy potentially enables the model to capture a richer set of semantic and contextual features as the condensed representations, bridging the gap between the language model's original training objective and the requirements for video encoding.
3.3 Adaptive Memory Selection
Through the streaming video encoding, it is feasible to use the encoded results from the final iteration, i.e,, HK. as a compact global memory that concludes the entire video. However, this fixed-length memory inevitably loses details, especially the information from early segments. Hence, this global memory alone is insufficient for comprehensive long video understanding.
To address this limitation, we make use of the encoded results of all historical clips of the input video, i.e., H = {HI, H2,... HR). Given a specific question or instruction, we first generate an adaptive indicator that summarizes relevant video content for that particular instruction. We accomplish this by reusing the language model in the streaming encoder, where we concatenate the global memory from the final iteration, HK, and the instruction texts, then pass the sequence into the model. We employ the output of the final token the instruction indicator, denoted as HQ RIID Thereafter, we calculate the cosine similarity between this instruction indicator and all historical clip indicators , H.,.., H E K and obtain the similarity distribution 8 E IR K To achieve a differentiable discrete selection, we adopt Humbel-Topk technique (36) produce binary index I that activates a subset of V out of K positions with the highest similarities:
= Gumbel- Tookll-TTo(s,,,.. V) E {0, 1)K
Based on I, we select the corresponding encoded results from H to formulate a subset of memories that are related to the instruction:
म (I+ H = 1},
where I denotes the selected indexes. We concatenate the selected memories H. in temporal order, resulting a sequence consisting of V x P tokens. Then, we feed the sequence with instruction texts into an LLM for comprehensive reasoning.
5

---

Table 1: The statistics of the av-
Table 2: Results on VideoChatGPT benchmark [0)]
erage video duration time of each
Method
Params
CI
DO
CU
TU
CO
evaluation dataset.
Video-LLaMA
88
7B
1.96
2.18
2.16
1.82
1.79
Dataset
Duration
VideoChat
7B
2.23
2.50
2.53
1.94
2.24
NexI-QA
79
42.23 sec
VideoChatGPT
50
7B
2.40
2.52
2.62
1.98
2.37
Next-GQA
80
39.60 see
MovieChat 66
7B
2.76
2.93
3.01
2.24
2.67
VideoChatGPT
1.81 min
LongVI.
75
7B
2.76
2.86
3.34
2.39
3.11
EgoSchema
51
3,00 min
ILaMA-VID
13B
3.07
3.05
3.60
2.58
2.63
MovieChat-IK
66
7.66 min
PILaVA
13B
3.27
2.99
3.66
2.47
3.09
MovieNet-QA
68
108.26 min
Ours
7B+1.3B
3.33
3.27
3.73
2.74
3.15
Our adaptive memory selection allows the model to dynamically access historical memories relevant to specific instructions, which mitigates the information loss inherent in the streaming encoding process. By drawing upon fine-gran fine-grained details across the full video duration, the LLM can provide detailed and informative responses, while preserving high computational efficiency.
3.4 Short-to-Long Training
To train VideoStreaming, we design progressive two-stage paradigm. First, we train single clip encoding on image and short video understanding tasks. Next, we train ceemory-propagatedd streaming encoding and adaptive memory selection as well as the LLM for long video understanding.
Single Clip Training. In this stage, both image- and video-text pairs are used to train the encoder to handle general visual signals. Following 44
88 43], we employ 790K image and short video caption data 64 O to train the MI.P projector for modality alignment. After that, we employ 763K image and video instruction data from
[50
to finetune the small language model. For video input, we uniformly sample T = 16 frames with spatial resolution 224 x 224 and use a frozen CLIP ViT-L/14 (5 to extract frame-wise features. After adjacent token merging, we obtain 16 x 64 = 1024 tokens as the clip feature representation. Then, the encoder, a two-layer MLP and small language model Phi-2 2.7B [20), distills each frame into P = 4 tokens, resulting in 16 x 4 = 64 tokens as the condensed representation with compression ratio of 16 1. For image-text pairs, we regard the images as single-frame clips and encode each into 4 tokens. We use standard next token prediction to consolidate visual contents into compact summarization tokens as illustrated in Fig. 2
50 45
Streaming Long Video Training. In the second stage, we use long video QA pairs to finetune the whole architecture, including VIT, the streaming encoder, and the LLM, shown in Fig. Ta The long video QA data encompasses three parts. (1) We adopt 25K movie QA pairs from 43 666. (2) We
curate a subset from Panda-70M [TT] which provides the original long videos and the captions of segmented clips. Based on this subset, we create 300K multi-round long video QA pairs with explicit timestamps. (3) We synthesize 20K long videos by concatenating short videos from existing QA datasets 76), and the original QA pairs correspond to different segments in the synthesized long videos. For each video, extract 16-frame clips 1 FPS, and the number of clips varies with the video duration. In streaming encoding, we employ the intermediate outputs from the first 16 layers of Phi-2 as the condensed memories. Finally, select V 4 most relevant timestamps and feed the selected memories of V x P = 256 tokens into the LLM, Vicuna-7B [33] for long video reasoning. Since our curated long video data could provide pseudo temporal grounding labels of specific questions, we utilize 30K QA pairs to warm up memory selection via a KL divergence loss. Subsequently,
the rest 315K QA pairs optimize the responses from the LLM and guide memory selection in a weakly-supervised manner. More training details are included in Appendix
Experiments
4.1
Datasets
We evaluate our model on long video QA datasets and present the statistics on the temporal duration of
individual datasets in Table. Among them, Next-QA QA [79], Next-GQA 880 and VideoChatGPT [50] encompass minute-long videos with thousands of frames. EgoSchema [1] contains over 5K three- minute videos with multiple-choice questions. Each question has long temporal certificate, requiring more than 100 seconds within video to produce a correct answer. MovieChat-IP 1K (66] and MovieNet- QA 68 consist of around ten-minute-long or even hour-long movies, posing significant challenges for the model to comprehend the visual contents across such long time spans.

---

Table 3: Results on the fullset test split of EgoSchema [ST].
Table Results on the validation set of Next-QA [79]. C,
T. D denotes causal, temporal and descriptive splits.
Method
Params
Fullset
Method
Params
D
All
finetuned
finetuned
VIC-VIT-L
7
424M
44.4
BLIP-2
39
4B
70.1
65.2
80.1
70.1
ViViT
55
1B
33.3
LLaMA-VOA 35
7B
72.7
69.2
75.8
72.0
Vamos
72
7B
72.6
69.6
78.0
72.5
zemossot
Intern Video 73
478M
32.1
zero-shor
FrozenBiLM
84
890M
26.9
Intern Video
478M
43.4
48.0
65.1
49.1
SeViLA
4B
22.7
SeViI.A 86
61.3
61.5
75.6
63.6
L.ooi 87
7B
33.5
Mistral
29
7B
51.0
48.1
57.4
51.1
Vamos 72
13B
36.7
L.ooVi 37
7B
55.6
47.9
63.2
54.3
LangRepo 32
7B
38.9
LangRepo
7B
57.8
45.7
61.9
54.6
LangRepo 32
8x7B
41.2
LangRepo
8x
64.4
51.4
69.1
60.9
Ours
7B+1.B
Ours
7B+1.3B
65.1
62.2
78.1
66.2
Table 5: Results on Next-GQA [8]
Acc @ GQA is defined as the percentage of questions that are
both correctly answered and visually grounded with loP 2 0.5.
Method
Params
mloP
IoP@0.5
mloU mloU@0.5
Acc@GQA
w/ specialized grounding module
TempCLIP [59]80]
130M
25.7
25.5
12.1
8.9
16.0
SeViLA 86
4B
29.5
22.9
21.7
13.8
16.6
w/o specialized grounding module
LLoVi
20.7
20.5
6.0
11.2
LangRepo
32
20.3
20.0
6.0
LangRepo
32
8x7B
31.3
28.7
18.5
12.2
17.1
Ours
7B+1.3B
32.2
31.0
19.3
13.3
17.8
4.2
Main Results
In this section, we present the results of our 8.3B model (half of Phi-2 2.7B in streaming encoder and Vicuna-7B the LLM). We omit the comparisons proprictary LLMs.
VideoChatGPT. Table presents the results on VideoChatGPT 50 in terms Correctness Infor- mation (CI), Detailed Orientation (DO), Contextual Understanding (CU), Temporal Understanding (TU) and Consistency (CO). Our model outperforms LLM-based video understanding methods on all five metrics, with a significant advantage in temporal understanding. memory-propagated streaming encoding architecture that explicitly captures temporal dynamics.
presents
be attributed the
the fullset test split of 7 consolidates a long-term memory to memorize long contexts but EgoSchema [51]. MC-ViT
EgoSchema. In Table
report the zero-shot performance
requires finetuning related dataset [19]. LLM-based methods [87 32 72
curate answers from
the captions of segmented video clips. However, these short-term captions cannot be optimized end-to-end and inevitably lose some detailed information. In contrast, encoder to produce memory embeddings in long videos and feed them into an L.MM to generate responses. Our model outperforms all zero-shot methods and comparable to the finetuned MC-ViT, demonstrating the effectiveness of our streaming architecture for long-term temporal modeling.
trainable streaming
Next-QA. In Table 4 perform zero-shor evaluation on the validation split of Next-QA [79] covering 5K multiple-choice questions. We respectively report the accuracy on Causal (C), Temporal (T) and Deseriptive (D) subsets. Our method consistently surpasses all zero-shot counterparts. Typically, compared to LangRepo (32] with Mixtral-8x 7B
(3), our 8.3B model improves the causal, temporal, and descriptive accuracy by 0.7%, 10.8%, 9.0% with considerably fewer model parameters.
Next-GQA. Besides the evaluation of the generated responses, we also assess the temporal grounding ability Next-GQA [80]. We calculate the Intersection of Prediction (IoP) and Intersection of Union (IoU), and use Ace @ GQA to measure the accuracy the correctly grounded predictions. According to the comparisons in Table 5 our simple similarity score based selection achieves the highest IoP and comparable loU to Se VLA 86 with a specialized grounding module. Moreover, the highest Acc @ GQA demonstrates the comprehensive capacity for grounding and high-level understanding.
MovieChat-IK. Table shows the results on MovieChat- [66), including a global mode for overall long-term understanding and breakpoint mode for detailed analysis of specific moments.

---

Table 6: Results on MovieChat- IK [66] global and breakpoint mode accuracy (Acc.) and score.
Gilobal mode
Becakpeint mode
Selocked
Sciadad
Global
Breakpoint
Method
Acc.
Score
Acc.
Score
1D
VideoChat
42
57.8
3.00
46.1
2.29
Vidco-LLaMA
88
51.7
2.67
39.1
2.04
VidcoChatGPT
50
47.6
2.55
48.0
2.45
MovieChat [66]
62.3
3.23
48.3
2.57
intervals Figure 4: Distribution of selected timestamps on
intenals
MovieChat+ 67
71.2
3,51
49.6
2.62
MovieChat-Ie We divide each video into multiple time intervals for statistical analysis.
Ours
90.4
4.42
54.9
2.80
Table 7: Results on MovieNet-QU QA 688
We present the used modality, the average number of tokens input to LLM and the average inference latency per question for comprehensive comparison.
Method
Text
Vision
Tokens
Latency
Overview
Plot
Temporal
-ID
43
18430
16.03 sec
3.09
3.31
2.02
MovieLLM 68
/
/
18430
16.48 sec
3.22
3.38
2.18
ILLMA-VID
43
X
/
5477
10,47 sec
2.28
2.88
1.46
M.
68
X
/
5477
10.43 sec
2.36
2.97
1.58
Ours
X /
256
5.32 see
2.65
3.13
1.88
In breakpoint mode, [66, 67 manually extract segments according to the timestamps in questions, while our model adaptively selects the related historical memories. Fig. D reveals that our selected timestamps are close to the ground-truths, and the higher breakpoint accuracy validates our adaptive selection effectively gathers the desired information from long contexts. Meanwhile, we reach significantly superior results in global mode, with the model's selection concentrated at the beginning and ending parts. On the one hand, the beginning of movie often contains hints of global information while the middle comprises redundant details. On the other hand, the condensed memories near the end of the video encapsulate the entire video, making them quite suitable for global understanding.
MovieNet-QA. Finally, we show the results on MovieNet-QA (68) consisting of 100 hour-long movies. Inspired by 168], we use GPT-3.5 produce scores in range 0-5 to evaluate the performance in overview, plot, and temporal understanding in Table 7 Specifically, LLaMA- VID [43] compresses each frame into two tokens, which are then combined with movie subtitles input to LLM. MovieLLM [68] further incorporates more generated data in training. These approaches largely rely on the texts for movie understanding, and only using visual frames leads dramatic performance drop. Moreover, its frame-wise compression is dependent on specific questions. The model has to reprocess the entire movie to extract visual features for different questions, resulting in high inference latency of over 10 seconds per question. Conversely, our architecture requires only once streaming encoding obtain a general condensed representation and adaptively selects significantly fewer tokens as input to LLM to answer specific questions. Therefore, we achieve a higher inference speed of 5.32 seconds per question and attain promising movie understanding without using subtitles.
Qualitative Results. We also present qualitative examples in Fig. 5 Typically, in Fig. 5a our model accurately captures the detailed descriptions in the question, and precisely selects the relevant segments that contain the corresponding character. Moreover, in Fig. (5b] given a two-hour long movie and a high-level question on the movie plot, without relying on subtitles, VideoStreaming can comprehend the intent of the question and select relevant scenes from the lengthy video. In particular, the model selects the scenes of tightrope walk, team disputes, and equipment setup, clearly illustrating the protagonist's challenges, thereby contributing to a comprehensive answer generation.
4.3 Ablation Study
We first explore four different settings on the memory and selection design of our model in Table g The details are as follows: (1) No propogated memory and no temporal selection, the model degener- ates into uniform sampling 64 frames for video understanding. (2) Without temporal selection, we adopt the memories of final 4 clips as the video-level representation. (3) Without propagated memory, we concatenate time indicators with questions to produce the question indicator to retrieve relative temporal segments. (4) The full architecture of VideoStreaming.
8

---

weuring white shirt mapping fingers during performance?
0-22.5 sec
43.0-56.3 sec
78.8990 sec
Ascistant:
likely member of band playing an singing along with the group. snapping his fingers keep the rhythm and add the overall sound the performance.
(a) example on Next- E... model accurately selects segments containing the target character.
challenges the protagonitt movie face goal?
25
45
83 min
98
Assistant: The protagonist the confronts of obstacles his endeavor accomplish remarkable foat. He devise method to access the unvering structures, tackle the complexities of setting the wire, maintain his equilibrium and concentration while suspendeed on tightrope, and monage the risks associated with installing the equipment on the towers.
(b) An example on a long movie. Our model selects typical segments that reveal the encountered challenges. Figure 5: Examples of question answering and the selected timestamps based on specific instructions.
Table Ablation studies the effects of memory selec-
Table 9: Ablation studies clip- and
tion and historical memory in streaming encoding.
frame-based sampling strategy.
Memory Selection
Fullset
Global Acc.
Break. Ace.
Sampling
Metric
Fullsct
MovieNet
X
34.4
52.5
21.6
Clip
Acc.
44.1
2.56
x
23.0
Frames
176
6032
X
/
38.4
43.8
39.1
44.0
2.61
44.1
90.4
54.9
Frame
Frames
180
6480
Historical Memory. In terms of the propagated memory in the streaming encoding process, i.c.,
H4-1 Eq 2 we report the fullset accuracy on EgoSchema [51] as well global and breakpoint accuracy on MovieChat-Il IK [66] in Table Typically, the historical memory significantly improves global understanding. This verifies our intuition that leveraging historical memory enables the model to produce a global representation that summarizes the entire video. Meanwhile, since we select small portion of the encoded results from the long video as input to LLM, the propagated memory across clips increases tolerance for imperfect temporal selection, as it preserves previous contexts. Without memory, there would be strict requirement on temporal selection accuracy to avoid completely losing necessary details, thus degrading the performance.
Memory Selection. We also validate the effects of our memory selection strategy. Comparing the results in Table. with respect to temporal selection, we have two observations. First, for breakpoint mode needing detailed understanding of specific moments, the lack of temporal selection leads to dramatic performance drop. It is crucial to select the related clips otherwise the LLM cannot catch the necessary details. Second, the historical memories in streaming encoding process enable the encoded results from the final iterations to provide coarse summarization of the entire video. Whereas, as shown in Fig. the beginning of the movie contains crucial cues for global understanding. Directly feeding the memories of final clips without temporal selection into LLM still results in information loss. These phenomena demonstrate the necessity of our adaptive selection for gathering detailed information over the long time span, which facilitates more accurate and informative responses.
Sampling Strategy.
default, we use clip-based sampling segment video into clips each consisting of fixed number of frames. Alternatively, it is also feasible to directly sample at moderate FPS, e.g.. FPS, and streamingly process the frames. Table showccses the comparison between clip- based sampling and frame- based sampling at FPS, and presents the average number of sampled frames on each dataset. The overall performance close, which verifies that our model can well generailze to different scenarios. And the longer MovieNer-QA videos, the frame-based sampling performs slightly better, as the clip-based approach limits the maximum number of clips, resulting in fewer total sampled frames for very long vidcos.

---

Table 10: Ablation studies on the streaming encoder architecture.
Encoder
Layers
Params
Fullset
Next-QA
Global Acc.
Break Acc.
MC-ViT
24
0.4B
32.3
53.1
71.2
40.4
Phi
4
0.3B
36.4
59.6
77.3
46.2
Phi
0.7B
39.8
63.2
84.3
49.2
Phi
I2
1.0B
42.5
65.1
87.4
51.2
Phi
16
1.3B
44.1
66.2
90.4
53.7
Phi
24
2.0B
43.8
66.0
90.0
53.7
Phi
32
2.7B
41.3
64.8
87.2
51.5
Vicuna
0.7B
39.8
63.2
84.3
49,2
Vicuna
1.3B
39.5
64.1
85.5
50.1
Vicuna
12
2.7B
40.2
64.4
85.7
50.1
Language Reasoning in Streaming Encoding. Besides, we ablate the necessity of using language model for memory consolidation. We compare our streaming encoder instantiated with partial layers of Phi-2.7B [20) with a conventional ViT-based approach, MC- ViT 77. We compare the results on EgoSchema fullset [51] and Next-QA 79, Global and Breakpoint Accuracy
MovieChat- 1K (66), as well as the number of encoder layers and parameters in Table TO It is clear that using language reasoning extract video memories exceeds the vision-based strategy even with fewer parameters. The reasons are twofold. On the one hand, MC-ViT updates memory according to handcrafted rules like clustering. In contrast, using language model the streaming encoder allows extensive video caption/QA data to guide memory consolidation an end-to-end manner. This data- driven strategy results more comprehensive memories that facilitate general video understanding. On the other hand, since the memory is fed into subsequent LLM for reasoning, it is casier to align the memory generated by language model with the input space of LLM, thus improving performance.
leverage
Streaming Encoder Architecture. Additionally, also explore alternative language models the streaming encoder. (1) A comparatively a small language model, Phi-2.7B (2), with different partial layers. (2) The previous layers of the large language model, Vicuna-7B (13]. Interestingly, using comparatively fewer layers of the language model leads to better results. We conjecture this is because the language model is originally trained for next token prediction. Its feature space of the final Transformer decoder layer might not align with the objective of visual content condensation.
Similar 82 46 17, the shallower layers might produce feature embeddings that encode richer information and serve as more comprehensive condensed video representations, Besides, under the same parameters, the smaller language model retains more layers and consistently outperforms the previous layers of Vicuna. Meanwhile, the prefix task for training the streaming encoder requires first training the entire model for next token prediction, before using partial layers in later stages. Therefore, using Phi as the streaming encoder brings another advantage in reduced computation.
More ablation studies temporal grounding supervision, the number of summarization tokens and selected timestamps, the time prompts, and the similarity measurement are included in Appendix
5 Conclusion
In this paper, introduce novel approach tackle the complexities of long video understand- ing with large language models (LL.Ms). Our proposed pemory-propagated streaming encoding architecture segments long videos into short clips and iteratively encodes each clip in sequence. By leveraging historical memory from preceding clips, we incorporate temporal dynamics into the encoding process and produce fixed-length memory encapsulate arbitrarily long videos. To further augment the detailed information for handling specific questions, we develop adaptive memory selection that selects relevant timestamps based on given instructions. This approach ensures that the most pertinent historical memories are utilized for question answering, thereby facilitating detailed and informative responses. Our model achieves superior performance with substantially fewer tokens and higher efficiency on extensive long video benchmarks. We demonstrate that memories from the streaming encoding significantly enhance global video understanding, while adaptive selection results in accurate temporal grounding with respect to specific questions.
10

---

Acknowledgement
This project is funded in part by Shanghai Artificial Intelligence Laboratory, the National Key R&D Program of China 22022DD0160000 the Hong Kong Generative Al Research and Development Center (HKGAI) under the Innovation and Technology Commission (ITC)'s InnoHK. Dahua Lin is Pl of HKGAI under the InnoHK.
References
[1] Sharegpt. hups://hharegpt.com 2023,
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprini xXXvv2333.08774 2023.
[3]
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc. Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716 1137166225 23736, 2022.
[4] Anas Awadalla, Irena Gao, Josh Gardner. Jack Hessel, Yusuf Hanafy. Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training
large autoregressive -aiignn- models. arXiv preprint arXiv:2308. 3930,, 2023,
[5] Jinze Bai, Shuai Bai, Yunfci Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint 2309v: 16609, 2023.
[6] Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. Frozen in time: joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1728- 1738, 2021.
[7]
Ivana Balažović, Yuge Shi, Pinclopi Papalampiii Rahma Chaabouni. Skanda Koppula, and Olivier Hénaff. Memory consolidation enables long-context video understanding. arXiv preprint arXiv:2402. 05861, 2024. Daniel Bolya, Cheng-Yary Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprins arXiv:2210.0944
[8]
2022.
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners.
Advances in neural information processing systems, 33:1877-1900 2020..
[10] Fabian Caba Heilbron. Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: large-scale video benchmark for human activity understanding. In Proceedings of the ieee conference on computer vision and pattern recognition, pages 961-970, 2015.
[11] Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Fun Jeon, Yuwei Fang, Hsin- Ying lee, Jian Ren, Ming-Hsuan Yang, et al. Panda-70m: Captioning 70m videos Hsin-Ying
with multiple cross- cross- modality teachers. arXiv preprint arXiv:2402.1944
2024.
[12] Ho Kei Cheng and Alexander G Schwing. Xmem: Long- video object segmentation with an atkinson- shiffrin memory model. In European Conference on Computer Vision, pages 640-658. Springer, 2022.
[13] Wei-L in Chiang, Zhuohan Li, 7i Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
[14] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Schastian Gehrmann, et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113,, 2023.
202.3
[15] Wenliang Dai. Junnan Li, Dongxu Li. Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv: (ato:::8800.0000 2018
[17] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang. and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19358-19999 1938-19969, 2023.
[18] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913, 2017.
[19] Kristen Grauman, Andrew Westbury, Eugene Byme, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 18995-19012, 2022.
[20] Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio César Teodoro Mendes, Allie Del Giorno Sivakanth Gopi, Mojan Javaheripi. Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. Textbooks are all you need, arXiv preprint 2306. 11044.1114,,0000

---

[21] Tengda Han, Weidi Xie, and Andrew Zisserman.
dense predictive coding for video representation learning. In European conference on computer vision, pages 312-329. Springer. 2020.
[22] Bo Hc, Hengduo Li, Young Kyun Jang, Menglin Jia, Xuefei Cao, Ashish Shah, Abhinav Shrivastava, and Ser-Nam Lim. Ma-Imm: eemory-auggmentee large multimodal model for long-term video understanding. arXiv preprint arXiv:2404.00 05726, 2024.
[23] Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus), arXiv preprint arXiv: 160.0.000.841
2016.
[24] Qingqiu Huang. Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. Movienet: A holistic dataset for movie understanding. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23 23-28, 2020, Proceedings, Part IV 16. pages 709-727. Springer. 2020.
[25] Suyuan Huang. Haoxin Zhang. Yan Gao, Yao Hu, and Zengchang Qin. From image to video, what do we need in multimodal llms? arXiv preprint arXiv:2404. 1865, 2024.
[26] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-worls visual reasoning and compositional question answering. In Proceddings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[27] Md Mohaiminul Islam, Ngan Ho, Xitong Yang. Tushar Nagarajan, Lorenzo Torresani, and Gedas Bertasius.
Video recap: Recursive captioning of hour-long videos. arXiv preprint arXiv:2402.1322. 13250, 2024.
[28] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Lee, Yun-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling visual and vision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904-4916. PMI.R, 2021.
[29] Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint 2..0xi::.300068880 2023.
[30] Albert Q Jiang. Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts. arXiv preprint arXiv:2401. 040088, 2024.
[31] Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv preprint arXiv:2311.08044 2023.
[32] Kumara Kahatapitiya, Kanchana Ranasinghe, Jongwoo Park, and Michael S Ryoo. Language repository for long video understanding. arXiv preprint arXiv:2403. /4622, 2024.
[33] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP). pages 787 0377799,222 2014.
[34] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya.
Reformer: The efficient transformer. arXiv preprint
arXiv:2001.00 04451,, 2020.
[35] Dohwan Ko, Ji Soo Lec, Wooyoung Kang, Byungscok Roh, and Hyunwoo Kim. Large language models are temporal and causal reasoners for video question answering. arXiv preprint arXiv:2310. 15747, 2023.
[36] Wouter Kool, Herke Van Hoof, and Max Welling. Stochastic beams and where find them: The gumbel- top-k trick for sampling sequences without replacement. In International Conference on Machine Learning, pages 349933999339 3508. PMLR, 2019.
[37] Ranjay Krishna, Yuke Zhu. Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz. Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-75 73, 2017.
[38] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv: 10t.220440..111 20221
[39] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-
inguage-imaag imaae pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19742. PMLR, 2023.
[40] Junnan Li. Dongxu Li. Caiming Xiong, and Steven Hoi. Blip: Bootstrapping maguage--maage pre- pre-training understanding and generation. In International conference on machine learning, pages 12888- 12900. PMLR, 2022.
for unified
[41] Jiapeng Li, Ping Wei, Wenjuan Han, and Lifeng Fan. Intentqa: Context-aware video intent reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11963-11974, 2023.
[42] KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu
Qiao. Videochat: Chat-centric vidco understanding. arXiv preprint arXiv:2305.06.33
2023.
[43] Yanwei Li, Chengyao Wang. and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language 2023.
models. arXiv preprint arXiv:2311.
[44] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:231 1. 10122, 2023.
[45] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint 2rX1::2310.03744, 2023.
12

---

[46] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024.
[47] Iya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on
Learning Representations, 2019.
[48] Ruipu Luo, Ziwang Zhao, Min Yang. Junwei Dong, Minghui Qiu, Pengcheng Lu, Tao Wang, and Zhongyu
Wei, Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306. 07207, 2023.
[49] Yun Luo, Zhen Yang, Fandong Meng, Yafu Ii, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning. arXiv preprint arX.v:2.308.0877 2023.
[50] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint urXiv:2306.0544 2023.
[51] Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form video language understanding, Advances in Neural Information Processing Systems, 36, 2024.
[52] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan I. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 11 11-20, 2016.
[53] Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Oer-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pages 947-952. IEEE, 2019.
[54] Scoung Wug Oh, Joon- Young Lee, Ning Xu, and Seon Joo Kim. Video object segmentation using space- time memory networks. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9226-9223 2225-9235,000 2019.
[55]
OpenAL. Introducing chatgpt, 2022.
[56]
OpenAI. Gpt4 technical report. arXiv preprint arXiv:2303.0877 2023.
[57]
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744, 2022.
[58] Pinelopi Papalampidi. Skanda Koppula, Shreya Pathak, Justin Chiu, Joe Heyward. Viorica Patraucean, Jiajun Shen, Antoine Miech, Andrew Zisserman, and Aida Nematzdeh. A simple recipe for contrastively pre-training vidco-first encoders beyond 16 frames. arXiv preprint ar arXiv:2312,00 073395,, 2023.
[59] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR, 2021.
[60] Alcc Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[61]
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer lcarning with unified text-to-text transformer. The Journal of Machine Learning Research, 21(1):5485-555
2020.
[62]
Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, and Lu Hou. Timechat: A time-sensitive multimodal large language model for long video understanding. arXiv preprint 2rXiv:2312.02051, 2023.
[63] Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A- okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pages 146-162. Springer, 2022.
[64] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2556-2565, 2018.
[65] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach. and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part 11 16, pages 742-758. Springer, 2020.
[66] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Yc, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token sparse memory for long vidco understanding. arXiv preprint urXiv:2307. 1.449, 2023.
[67] Enxin Song. Wenhao Chai, Tian Ye. Jenq-Neng Hwang. Xi Li, and Gaoang Wang. Moviechat+: Question- aware sparse memory for long video question answering. arXiv preprint arXiv:2404. /7176, 2024.
[68] Zhende Song, Chenchen Wang. Jiamu Sheng, Chi Zhang, Gang Yu, Jiayua Fan, and Tao Chen. Moviellm: Enhancing long video understanding with ai-generated movies. arXiv preprint arXiv:2403.00 0014222 2024.
[69] InternLM Team. Internlm: A multilingual language model with progressively enhanced capabilities. https: / /g thub. com/ Intpe/nntettteeeeeeeeeeeeeeeeert Bwwpel/MMIineertteeeeee InternLM 2023.
13

---

[70] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
[71] Jiahao Wang, Guo Chen, Yifei Huang, Limin Wang. and Tong Lu, Memory-andd-atiiipaaii transformer for online action understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13824- 13835, 2023.
[72] Shijie Wang. Qi Zhao, Minh Quan Do, Nakul Agarwal, Kwonjoon Lee, and Chen Sun. Vamos: Versatile action models for vidco understanding. arXiv preprint arXiv:2311. 13627, 2023.
[73] Yi Wang. Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. Internvideo: General video foundation models via generative and discriminative
learning. arXiv preprint arXiv:22/2.03399 2022.
[74] Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Lc. Finetuned language models are zero-shot leamers. arXiv preprint 2021.
[75] Yuctian Weng, Mingfei Han, Haoyu Hc, Xiaojun Chang, and Bohan Zhuang. Longvlm: Efficient long vidco understanding via large language models. arXiv preprint trXiv:2404.03366 2024.
[76] Bo Wu, Shoubin Yu, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. Star: A benchmark for situated reasoning in real-world videos. In Thirty-fifth conference on neural information processing systems datasets and benchmarks track (Round 2). 2021.
(77] Chao- Yuan Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. Long-term feature banks for detailed vidco understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 284-293, 2019.
[78] Chao- Yuan Wu, Yanghao Li, Karttikeya Mangalam, Haoqi Fan, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Memvit:
multiscale vision transformer for efficient long-term vidco recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13587-13597, 2022.
[79] Junbin Xiao, Xindi Shang, Angela Yao, and Tat- Seng Chua, Next-qa: Next phase of question-anae answering to explaining temporal actions. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pages 9777-9786, 2021.
[80] Junbin Xiao, Angela Yao, Yicong Li, and Tat Seng Chua. Can trust your answer? visually grounded video question answering. arXiv preprint arXiv:2309.01322 2023.
[81] Jun Xu, Tao Mci, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288-5296, 2016.
[82] Jiarui Xu and Xiaolong Wang. Rethinking self-supervised correspondence learning: A video frame-level similarity perspective. Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 10075-10085, 2021.
[83] Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404. 16994, 2024.
[84] Antoine Yang. Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. Advances in Neural Information Processing Systems, 35:124-141, 2022.
[85] Qinghao Ye. Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, al. mplug-owl: Modularization empowers large language models with multimodality, arXiv preprint arXiv:2304. 14178, 2023.
[86] Shoubin Yu, Jaemin Cho, Prateek Yadav, and Mohit Bansal. Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems, 36, 2024. [87] Ce Zhang, Taixi Lu, Md Mohaiminul Islam, Ziyang Wang, Shoubin Yu, Mohit Bansal, and Gedas Bertasius. A simple Ilm framework for long-range video question-answeringg arXiv preprint arXiv: 17235, 2023. [88] Hang Zhang, Xin Li, and Lidong Bing. Video-Ilama: An instruction sortucttio-uteedd audio-visual language model for video understanding. arXiv preprint arXiv:2306.0288.
2023.
[89] Xingyi Zhou, Anurag Amab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. Streaming dense video captioning. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 18243-1822 18252, 2024.
[90] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:230..1059. 2023.
14

---

Limitations
One potential limitation is that we simply uniformly sample frames form set of short clips for eecmory-propageen streaming encoding. However, in a long video, different segments possess different amounts of information. The uniform sampling may result in using redundant tokens for clips with bland content. Meanwhile, the number of tokens used to represent clips with abundant visual contents and intensive temporal dynamics may be insufficient, leading information loss. To address this limitation, we plan to explore adaptive segmentation techniques that dynamically adjust the segmented clip lengths based on the complexity and content of the video.
Impact Statements
Our proposed VideoStreaming, a streaming long video understanding architecture with large language models has various potential impacts for society. On the positive aspect, VideoStreaming contributes to improved intelligent video understanding, especially for long videos. This could be beneficial in education, entertainment, and information retrieval, where users often need to navigate and understand complex video materials. Besides, our technique could lead to advancements in multimedia analytics with applications in areas like video surveillance, market research, and content personalization.
On the negative aspect, the ability to efficiently process and retrieve information from long videos raises potential privacy and security concerns. If misused, this technology could be employed for unauthorized surveillance, personal monitoring, other unethical purposes that infringe individual privacy, addition, the enhanced video understanding capabilities might be exploited for the creation of manipulated or misleading video content, leading to the spread of misinformation and the potential for social manipulation.
In conclusion, despite that VideoStreaming presents advancement in long video comprehensive, its development should be accompanied by careful consideration of ethical and societal implications.
A More Implementation Details
We use CLIP ViT-L/14 [59] to extract frame-wise features with input resolution 224 224, resulting in
256 tokens per frame. Then, we concatenate every four spatially adjacent visual tokens along channel dimension, representing each frame with 64 tokens with channel dimension 4096. The streaming encoder consists of a two-layer MLP projector (channel dimension 4096-2560-2560) with GELU activation 23 and language model Phi-2 2.7B 20). In the first training stage, we initially freeze Phi- 2. and only tune the MLP projector on 790K caption pairs, including 558K image caption data from CC3M (64] and 232K short video caption data from WebVid 2.5M (6. Following LLa [46, 45). we use Adam W optimizer [47] with global batchsize 256, initial learning rate x 10 with cosine decay to train epoch for modality alignment. Subsequently, we jointly train Phi-2 and the MLP
projector on 763K QA pairs, including 625K image QA pairs [18] 26 33, 37 46, [52 [3] 63 65), 40K text conversations
① and 98K video QA pairs 10, with global batchsize 128, initial learning rate 2 x 10-5 5 with cosine decay.
In the second stage, we jointly train ViT, the streaming encoder and the LLM on long video data. In the ceenory-propagaterr streaming encoding process, we insert a brief prompt to indicate the explicit timestamps of the historical memory and the input clip formulated as This contains a history of (start] to (end) seconds, and a clip sampled in [start] to (end) seconds.. We adopt the output of the first 16 layers out of the 32 layers of Phi-2 as the condensed representation. Then, we adaptively select 4 most relevant timestamps and feed the associated 256 memory tokens into a two-layer MLP projector with channel dimension 2560-4096-4096 and an LLM, Vicuna-7B [13] generate the final responses. We jointly train the whole architecture, including Vicuna, Phi-2, MLP projectors and ViT encoder, on long video QA data with global batchsize 128, initial learning rate 10- with cosine decay. In default, we first use 20K synthesized long videos and sample 10K QA pairs curated from Panda-70M with pseudeo temporal grounding labels to train memory selection as warm-up. The learning objectives contain a standard next token prediction loss and supervised KI divergence loss that aligns the distribution of the predicted memory selection results and the pseudo temporal labels. Next, based on the warm-up model, we further train the rest 295 long video QA pairs only with
15

---

Data provided by Panda- 70M:
[1, 12]: The speaker is a fomale news reporter who is sitting in front of studio camera.
[15, 233: group of people standing in line outside of building.
(25, 28): Two children standing next to each other on to screen.
1153, 161f: A woman is interviewed by reporter in park.
Created Q4 pairs:
Q: Describe the video in chronological order with detailed timestamps.
A: In I to 12 seconds, the speaker is female reporter who is sitting in front of studio camera. In 15 23 seconds, group of people standing in line outside of building. In 25 to 28 seconds, two children standing next each other screen, ... In 162 166 seconds,
interviewed reporter in park.
Q: Summarize the events in 15 to 23 seconds.
group of people standing line outside building.
Q: What timestamp that 104 seconds.
in suit and talking microphone press conference?
What happens
64 seconds? picture a person and their family from the featured in video.
Figure 6: An example of the QA pairs from the captions and segmented timestamps from Panda- 70M [Л]]
Table 11: Results on the test set of IntentQA 41
Method
Params
Why
How
Before/
All
LLaMA-ViL VID
40.1
36.3
41.4
LLoVi 87
7B
57.9
55.4
42.3
53.6
LangRepo
32
7B
56.9
60.2
42.1
53.8
LangRepo
32
8x7B
62.8
62.4
47.8
59.1
Ours
7B+1.3B
65.6
66.2
59.0
64.1
next token prediction loss. The whole training conducted on 32 A100 (80G) GPUs for around 2.5 days.
B Long Video QA Data Creation
In addition to the existing 25K long video QA pairs movies [66 [3), create more QA data from two aspects. First, we leverage the existing short video QA dataset [76 8T and synthesize short videos into minute-long videos with average duration of one minute. The original questions of each short video coarsely correspond to a temporal segment in the synthesized long video. We use this correspondence noisy labels supervise the memory selection. Second, recent Panda-70M [II) segments long videos into short clips and produces captions for each clips. This dataset provides the original long videos, the captions of segmented clips as well the segmentation timestamps. Based on these cues, we produce multi-round QA conversations. Below we show an example in Fig 6 The produced time-sensitive QA pairs are crucial enhance the temporal awareness and guide precise memory selection in long videos.
C More Experimental Analysis
Quantitative Results on IntentQA. IntentQA 41 is long-form video understanding dataset consisting of 4.3K videos with 16K multiple-choice questions, which are classified into three types, why, how and before/after. In Table [1] we report the zero-shot performance on IntentQA test set. Our method presents dominant advantage in temporal understanding. And the overall performance is significantly superior to recent works.
16

---

Feature Similarity
Temporal distance
0.19
0.15
608
Ratios
Ratios
606
0.10
0.5
0.3
0.7
17
Cosine similarity
Time intervals
Figure Visualization of the feature similarity and temporal distance of selected clips.
Table 12: Ablation studies on the use of temporal grounding supervision. Acc denotes the ratio of correctly answered ques- tions on NexI-GQA [80 regardless of grounding accuracy.
Watn up
Worts s-iiieed
Raliox
0.10
Warm-up
Mixed
Fullset
mloP
mloU
Acc@GQA
Acc
000
x
43.7
24.1
9.8
11.1
54.9
x
44.1
32.2
19.3
17.8
55.7
Time intervals
X
/
43.9
28.5
14.6
15.4
55.3
Figure 8: Distribution of selected times-
/
/
44.0
32.1
20.0
17.7
54.8
tamps Next-GQA.
Analysis on the Selected Clips. To verify whether the selected clips are redundant, we provide statistics on the feature similarity and the temporal distribution of the selected clips in Fig. 7 For feature similarity, we calculate the cosine similarity the time indicators of the selected clips. We divide the x-axis from 0 to into 20 bins, and it shows the distribution of feature similarity. The average cosine similarity 0.68. For temporal distance, we calculate the time intervals between the selected clips (represented as the ratio of the total video duration) and visualize the distribution of these time distances. The average distance is around 35% the total vidco length. The statistics on the feature similarity and temporal distance indicate that the selected clips not redundant.
D More Ablation Studies
We provide more ablation studies on the use of temporal grounding supervision, the number of summarization tokens and selected timestamps, the effects of time prompts in the memory- proppaated streaming encoding process, and the similarity measurement used in memory selection.
Temporal Grounding Supervision. First, we present the studies on the use of temporal grounding supervision. As mentioned in Section 3.4 we employ around one-tenth of long video QA pairs to provide pseudo temporal labels. We compare four training strategies: (1) Fully weakly-supervisee manner without any pseudo labels. (2) Using pseudo labels to train a warm-up model, then expanding to large-scale QA pairs. (3) Mixing all long video QA data, where the model uniformly receives temporal supervision in training. (4) Training mixed data after warm-up initialization. The results on EgoSchema [5T] and Next-GQA [80] in [bbbeei2iiddcatt three key points: First, warm-up training contributes to more powerful grounding ability. The sparse temporal label supervision in mixed mode is overcome by the powerful initialization from warm-up training, which can generalize to large-scale data. Second, reusing the temporal labels after warm-up offers no additional benefits, so we adopt warm-up as the default setting. Third, without using temporal labels, the grounding performance drops, but the QA accuracy remains stable. Fig. g reeealss that compared to those trained with temporal labels, the weakly-supervisee model selects relatively later segments that preserve previous contexts with the help of historical memory, thus maintaining comparable QA capacity.
The Number Summarization Tokens and Selected Timestamps. compare using different number of summarization tokens and selected timestamps, i.c., P in Eq. and V in Eq. We
17

---

Table 13: Ablatoin study on the number of summarization tokens and selected timestamps. We report the results on EgoSchema [T and Next-GQA [0]]
V
Tokens
Fullset
Acc@ GQA
4
64
32.1
9.8
8
128
33.4
10.5
4
1
64
41.6
15.5
256
44.1
17.8
4
8
512
44.9
18.0
256
42.5
16.3
16
4
1024
43.8
17.9
Table 14: Ablation study on the formulation of time prompts.
We report the results on
EgoSchema [5I] Next-GQA (80) and MovieChat-IK [66].
Prompt
Fullset
Acc@GQA
Global Accuracy
Breakpoint Accuracy
None
38.8
12.4
66.7
19.8
Clip
40.5
15.4
70.3
44.5
Memory
42.1
16.1
83.3
43.1
Clip+Memory
44.1
17.8
90.4
54.9
compare the performance as well as the number of tokens input to LLM in Table We conclude three observations. First, too few summarization tokens, e.g., P 1, leads to substantial performance drop, since condenses a 16-frame into only 16 tokens with significant information loss in spatial contexts. Such information loss cannot be compensated by selecting
temporal segments. Second, the performance saturates when improving P from to 16. This is because the existing video benchmarks [51] [0] do place high demands on spatial detail understanding. It is sufficient to represent each frame with 4 tokens on average. Third, increasing the number of selected timestamps only results in minor improvements, which is not proportional to the increased number of tokens. This can be attributed to the historical memory used in the streaming encoding process. The utilization of historical memory enables the condensed representation of each clip to encompass the information in preceding clips, which enlarges the temporal receptive field. Hence. increasing the number of selected timestamps does not proportionally increase the temporal receptive field, resulting in slight performance improvements.
Time Prompts. We explore three different formulations of the time prompts used in memory- propagated streaming encoding: (1) Only with the timestamps of the current clip, e.g., This clip is sampled in |start] [end] seconds. (2) Only with the timestamps of the historical memory, e.g.. This contains history of [start] to (endj seconds. (3) Simultaneously with the timestamps of the historical memory and the current clip, e.g., This contains a history (start) to (end) seconds, and clip sampled in /start) to /end) seconds. We report the results of different time prompts in Table [4] is obvious that the lack of time prompts leads to substantial performance drop in the MovieNet- IK breakpoint mode accuracy, which requires detailed analysis of specific moments. The reason is that the breakpoint mode requires the model to answer questions specified timestamps, the time prompts provide the model with necessary information in adaptive selection. Meanwhile, incorporating the timestamps of historical memories results more significant improvements in global understanding. Overall, jointly leveraging the memory and clip timestamps contributes to the best results.
Table 15: Ablation study on the similarity measurement. We report the results EgoSchema 51
Next-GQA [0) and MovieChat-I K [66]
Similarity
Fullset
Ace@ AQA
Global Accuracy
Breakpoint Accuracy
Cosine 44.1
17.8
90.4
54.9
Dot product
34.5
9.3
55.6
22.1
18

---

Table 16: Ablation study different settings, including memory propagation temporal selection, the number of summarization tokens and the number of selected clips. on hour-long MovieNet- QA benchmark from three perspectives.
Memory
Selection
P
V
Overview
Plot
Temporal
1.98
2.23
1.39
X
16
4
2.51
2.61
1.63
16
4
2.24
2.77
1.52
V
16
4
2.65
3.13
1.88
V
4
4
2.53
2.82
1.73
V
4
8
2.58
3.02
1,83
16
8
2.68
3.17
1.95
Similarity Measurement. Besides, present the study on the similarity measurement used in adaptive memory selection. We compare the default cosine similarity against simple dot product without normalization in Table 15| Empirically, observe that dot product could result in numerical instability, leading to overflow in training. Consequently, the calculated similarity score cannot reflect the correlation between the instruction and different segments and results in poor results questions that require accurate temporal grounding, e.g., the Acc GGA metric on Next-GQA [0) and the breakpoint mode accuracy MovieChat-1K [66]
Finally, we show the ablation studies of different settings particularly the hour-long video benchmak, MovieNet-QA in Table is clear that for hour-long videos, the propagated memory is crucial for understanding the overview contents, detailed plots as well as temporal relations. Meanwhile, the temporal selection significantly improves detailed plot analysis, and appropriately increasing the number of selected clips can further improve performance.
Analysis on Long Video Benchmark.
19

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state contributions, and the claims match the experimental results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Section 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Detailed instructions for replicating the results are provided in Section 3.4 and Appendix A Additionally, the code, model checkpoint, and data will be publicly released.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
21

---

Answer: [No]
Justification: The creation of long video QA data is presented in Appendix Detailed information to reproduce all experimental results is provided in Section 3.4 and Appendix And the code and data will be released upon paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.co cc/
jublic/guidess Ttaildes/oodeSuumissi....oo cot.isssyinooyyyl for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/mips/cccpublic//uu.ide httpc//guiidee/oeeuubbbbbiii
ssiiisosicyyyyyc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
The experimental setting is presented in Section 3.4 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
22

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information on the computer resources is provided in Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://aarr....c.ooo......ccttta
Answer: [Yes]
Justification:
The research adheres to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The potential impacts are discussed in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: We will require to adhere to specific usage guidelines to access the model and datasets, ensuring that they are used responsibly and to mitigate the risk of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have properly credited the creators or original owners of assets used in the paper and we use the license CC-BY 4.0.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
24

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
All new assets introduced in this paper will be well documented upon their release.
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
25


Reviews:
[{"review_id": "PuYAoh5XPO", "reviewer": "Reviewer_FUM4", "summary": "In this paper, a vision-language model, named VideoStreaming, is proposed. \nIt adopts the memory based architecture to understand long video. Specifically, it segments a long video into multiple short clips and encodes each clip sequentially. During this encoding process, it exploits the memory feature to propagate the information from previous clips. Also, it develops the memory selection strategy to find out question-related memory among all encoded memories.\nExtensive experimental results on various VQA datasets show that the proposed algorithm achieves better performances than exisiting methods.", "questions": "In overall, I think that the motivation of the paper is reasonable. Also, the proposed memory propagating encoding framework and memory selection method have some contribution to the field. Therefore, I'm positive about this work. Please find my concerns in the weakness section.", "limitations": "Yes, L574-581", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is well-written and easy to understand\n- The motivation is reasonable. Also, the proposed algorithm technically sounds.\n- The proposed algorithm achieves", "weaknesses": "I could not find the critical flaws in this work. \nEven though the motivation is not very new in long video understanding, the proposed memory-based framework is reasonable and well-designed.\n\n- It would be better to provide the comparsion on IntentQA since it is also widely used for LVQA.\n- In L201, the authors said 'we develop a variant of Gumbel-Softmax, denoted as Gumbel-TopK(),...' However, there is no detailed description on Gumbel-TopK in the paper and Gumbel-TopK trick is already widely used. Thus, I would recommend using other expression, instead of 'develop.' Otherwise, it would be better to explain Gumbel Top-k in detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for recognizing our motivation, technical algorithm, and writing. Below we would like to provide point-to-point responses to all the raised questions:\n\n> **Q1: The comparison on IntentQA dataset.**\n\n**A1:** We compare our model with recent advanced methods in the Table below. We report the zero-shot performance on IntentQA test set. Our method presents a dominant advantage in temporal understanding. And the overall performance is significantly superior to recent works.\n\n| Method | Params | Why | How | Before/After | All |\n| :-: | :-: | :-: | :-: | :-: | :-: |\n| LLaMA-VID | 13B | 43.8 | 40.1 | 36,3 | 41.4 |\n| LLoVi   | 7B  |   57.9  | 55.4  | 42.3 | 53.6 |\n| LangRepo   | 8x7B   | 62.8 | 62.4 | 47.8 | 59.1 |\n| Ours | 7B+1.3B | 65.6 | 66.2 | 59.0 | 64.1 |\n\n> **Q2: The details on Gumbel-TopK.**\n\n**A2:** Sorry for the confusion and missing the detailed explanation for Gumbel-TopK. Similar to the Gumbel-Softmax technique, Gumbel-TopK is used to allow gradient backpropagation when selecting the Top-K indices. The major difference is that Gumbel-TopK can take multiple indices (i.e., the Top-K largest activations) for each sample, while Gumbel-Softmax only takes the index corresponding to the single largest activation for each sample. This design allows us to select multiple video segments that are relevant to the questions (i.e., the Top-K most related ones) to do comprehensive reasoning. This is not a technical contribution, and we will modify 'develop' into 'adopt' for correction.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GY66v3oydA", "reviewer": "Reviewer_QjSL", "summary": "This paper presents an advanced vision-language model designed to handle the complexities of understanding arbitrary-length videos. The model addresses the computational challenges posed by long video sequences by implementing two core techniques: Memory-Propagated Streaming Encoding and Adaptive Memory Selection. These innovations enable the model to efficiently encode long videos into a constant number of video tokens, preserving both spatial and temporal dynamics. Extensive experiments demonstrate that VideoStreaming achieves superior performance and higher efficiency on long video benchmarks, showcasing its ability to provide precise temporal comprehension for detailed question answering. The contributions include a novel approach to long video understanding, efficient encoding and memory selection strategies, and comprehensive evaluations that highlight the model’s capabilities and advantages over existing methods.", "questions": "- Performance on Egoschema: I am very curious about the performance on the Egoschema dataset. As it is an egocentric video dataset, and since the pretraining dataset does not include egocentric videos, how does the model achieve a significant zero-shot performance (44.1) comparable to the finetuned MC-ViT-L baseline (44.4)? Could you provide more insights or experiments that explain this result?\n\n- Visualization in Figure 5(b): The accurate grounding along a video exceeding one hour is impressive. However, I wonder if the LLM already recognizes specific information from the query, such as \"Philippe Petit\" and \"Twin Towers.\" Additionally, the grounding intervals and the text outputs do not seem to have solid associations. Could you clarify how the grounding intervals are determined and ensure that they are directly linked to the text outputs? Are there any qualitative or quantitative measures used to validate these associations?", "limitations": "Please refer to the weakness & questions section.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper addresses a significant and challenging research problem: long video understanding.\n- The proposed strategy is technically sound.\n- The paper conducts sufficient experiments on varied benchmarks, utilizing six video datasets of different durations.\n- The use of an additional small encoder, such as Phi-2, as a streaming encoder is an interesting approach.", "weaknesses": "- Summarization Tokens: The use of summarization tokens is unclear, as it avoids attention between the TT tokens and the clip TN tokens. This design choice needs better justification and clarification.\n\n- Small Language Model for Streaming Encoder: While the use of a small language model for the streaming encoder is interesting, there are no experiments demonstrating the necessity of language reasoning for visual-dominant dependencies (mainly visual embed. outputs). Additionally, the model is still quite large (2.7 billion parameters). A more efficient alternative, such as a shadow layer transformer, should be considered.\n\n- Paper Writing: The proposed technical strategy is difficult to understand and lacks readability. It would benefit from clearer explanations and more detailed descriptions.\n\n- Memory-Propagated Strategy: The memory-propagated strategy is similar to previous strategies in video understanding, such as those in [1]. The paper should differentiate its approach more clearly.\n\n[1] Memory-Augmented Dense Predictive Coding for Video Representation Learning\n\n- Ablation Studies: The ablation studies are not comprehensive, with only Table 8 presented. There is a lack of convincing studies on several designs, such as summarization tokens. More detailed ablation studies are needed to validate the design choices.\n\n- Baseline Comparisons: In the main tables, the paper does not include recent instruction-tuning baselines such as LLaMA-VID. Many baselines, like LangRepo and LLoVi, are mainly modular methods based on image captioners like LLaVA, making the comparisons unfair. Including more relevant and recent baselines would provide a clearer picture of the paper's contributions.\n\n- Progressive Training: The progressive training strategy is not innovative and can be found in previous works [1, 2]. The paper should highlight any unique aspects of its approach.\n[1] LLaMA-VID\n[2] A Simple Recipe for Contrastively Pre-training Video-first Encoders Beyond 16 Frames", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for acknowledging the significance of the research problem, the interesting and sound technical design, as well as sufficient experiments. Below we provide point-to-point responses to all the raised questions:\n\n> **Q1: The use of summarization tokens.**\n\n**A1:** Our motivation is to distill the information in a video clip (TN clip tokens) into a compact set of tokens (TP summarization tokens). These TP tokens are intended to serve as a memory that summarizes the clip content, which can be propagated across different clips for long-term memory formulation.\n\nTo achieve this, we initialize TP summarization tokens using pooling and feed them into an autoregressive language model. The causal attention layers then aggregate the clip information into the TP tokens to create a compact memory. To ensure these tokens consolidate useful information, we design a prefix task where the language model generates captions using only the TP tokens, without access to the original clip features. This avoids attention between the TT caption tokens and the TN clip tokens, encouraging the TP tokens to encapsulate the clip information.\n\n> **Q2: The necessity of language reasoning for visual-dominant dependencies.**\n\n**A2:** We validate the necessity of language reasoning in Table 2 in rebuttal PDF. We reimplement a conventional video memory strategy MC-ViT [7] for comparison. Our method presents dominant advantages and the reasons are two-fold. \n+ MC-ViT consolidates memory according to handcrafted rules like clustering. In contrast, using language model as the streaming encoder allows us to leverage extensive video caption/QA data to guide memory consolidation in an end-to-end manner. This data-driven strategy results in more comprehensive memories that facilitate general video understanding. \n+ Since the memory is fed into a subsequent LLM for reasoning, we need to align the memory with the input feature space of the LLM. Compared to the output from ViT backbone, it is easier to align the memory generated by a language model with the feature space of LLM, thus improving performance.\n\n> **Q3: More efficient alternatives in streaming encoder.**\n\n**A3:** We try fewer layers of Phi model as streaming encoder in Table 2 in rebuttal PDF. It indicates too few layers would lead to performance drop due to insufficient memory consolidation. However, even with fewer parameters (0.3B with 4 layers), the language model-based approach still outperforms MC-ViT, which reinforces the necessity of language reasoning for visual feature processing.\n\n> **Q4: The difference in memory propagation with MemDPC.**\n\n**A4:** The major difference lies in the memory content, the memory usage and the technical design. \n+ MemDPC formulates a learnable memory bank shared for the entire dataset. The memory serves as the statistics of the whole dataset. In contrast, our memory is a summarization of a specific video.\n+ The memory bank in MemDPC is an external knowledge reference that provides multiple future hypotheses for future prediction. Our memory is used to conclude the historical information in a long video to facilitate long-term comprehension.\n+ MemDPC only uses memory to augment the future prediction, there is no explicit memory propagation process. Conversely, in our architecture, the memory is propagated across different clips to form a global encapsulation of a long video.\n\n> **Q5: More ablation studies.**\n\n**A5:** Besides Table 8\\~9, we have presented ablation studies on summarization tokens, temporal supervision, time prompts, and similarity measurements in Table 10\\~13 in the appendix (Page 15\\~17).\n\n> **Q6: More baselines like LLaMA-VID.**\n\n**A6:** We add the results of LLaMA-VID with Vicuna-13B on EgoSchema, Next-QA and MovieChat in Table 3 in rebuttal PDF. For MovieChat breakpoint mode, we only input segments of the video up to the breakpoint timestamp to the model. It is easier than our setting which requires adaptive temporal selection. Nevertheless, our model still significantly outperforms LLaVA-VID on all benchmarks.\n\n> **Q7: About progressive training.**\n\n**A7:** Sorry for the confusion. The progressive training part is used to clarify the training process rather than serve as an innovation. We will modify this to make it clearer.\n\n> **Q8: Explanations for significant zero-shot performance on EgoSchema.**\n\n**A8:** The reasons are two-fold. \n+ The questions in EgoSchema do not have obvious ego-centric domain-specific characteristics, but rather require long-term temporal memory and perception, which is a major strength of our model.\n+ Compared to MC-ViT, our architecture with LLM has stronger generalization ability. It can effectively generalize to different video scenarios even without ego-centric training pairs. This is also validated by the superior zero-shot performance of LLaMA-VID 13B compared to prior methods like InternVideo and FrozenBiLM (35.5 vs 32.1, 35.5 vs 26.9).\n\n> **Q9: About grounding in Figure 5.**\n\n**A9:** For grounding, we calculate the similarity between the question and the time indicators of each clip to select the related segments. Below are some explanations.\n+ The similarity-based strategy relies on some semantic relations between questions and clips to do temporal selection. It can correlate the key components in questions like Tower with the clips containing corresponding elements.\n+ The grounding intervals are determined by the temporal length of the sampled video clips.\n+ We only feed the encoded memories of the selected clips to LLM, so the model fully relies on the grounding results to produce responses.\n+ For quantitative evaluation, Next-GQA requires accurate grounding to produce correct answers. The results in Table 5 in the manuscript validate our method reaches promising grounded question-answering ability, even exceeding the methods with specialized grounding modules. There currently lack hour-long grounding QA benchmarks for further evaluation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FJPcywM9qC", "reviewer": "Reviewer_9Ye5", "summary": "This paper introduces VideoStreaming, a Vision-Language Large Model (VLLM) designed for comprehensive video understanding. It effectively manages the challenges of computational overhead associated with processing long videos by innovatively using a constant number of video tokens. VideoStreaming demonstrates superior performance and efficiency on long video benchmarks, excelling in detailed question answering with accurate temporal comprehension.", "questions": "Please see weaknesses. I will raise my score if the author well address my concerns.", "limitations": "None.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The Memory-Propagated Streaming Encoding technique effectively captures long-term video content while maintaining a manageable computational footprint by propagating memory through sequential segments.\n2. Adaptive Memory Selection strategically reduces the number of tokens used in processing, selecting only the most relevant memories for generating responses, which enhances the model's efficiency and precision.\n3. Demonstrated superior performance on long video benchmarks indicates the model's capability to handle complex video understanding tasks effectively, setting it apart from existing models.", "weaknesses": "1. The core strategy still relies on processing video through key clips, which missed some relevant works, such as \"LGDN: Language-guided denoising network for video-language modeling\" (NeurIPS 2022).\n2. The method for selecting key clips could potentially lead to redundancy, especially if multiple similar clips have high relevance scores. Integrating a regularization function to promote diversity in key clip selection could help mitigate this issue.\n3. In scenarios involving multiple queries about a video, the model may need to repeatedly extract key clips for each question, potentially affecting efficiency. The paper could benefit from a discussion on optimizations that reduce the computational overhead in such use cases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for acknowledging our model's ability to handle long videos with high efficiency. Below we would like to provide point-to-point responses to all the raised questions:\n\n> **Q1: Related work on processing key clips.**\n\n**A1:** Thanks for pointing out this related work. LGDN is a pioneering work in selecting the key clips that are highly related to the text prompts. It employs a contrastive learning architecture to learn global video-text correspondence and then uses the model to calculate fine-grained clip-text correspondence scores to filter out noisy frames. The key difference is that we can leverage both fine-grained temporal grounding labels and implicit gradient flow from the next token prediction loss to guide the learning of temporal selection. We will add the discussions in the related work.\n\n> **Q2: The redundancy and diversity of selected clips.**\n\n**A2:** Thanks for the promising suggestion of integrating a regularization term in the key clip selection stage. In the author rebuttal PDF, we provide statistics on the feature similarity and the temporal distribution of the selected clips in Figure 2.\n+ For feature similarity, we calculate the cosine similarity of the time indicators of the selected clips. We divide the x-axis from 0 to 1 into 20 bins, and it shows the distribution of feature similarity. The average cosine similarity is 0.68.\n+ For temporal distance, we calculate the time intervals between the selected clips (represented as the ratio of the total video duration) and visualize the distribution of these time distances. The average distance is around 35% of the total video length. The statistics on the feature similarity and temporal distance indicate that the selected clips are not redundant. \n\nWe will explore integrating a regularization term in future work.\n\n> **Q3: The efficiency of key clip extraction with multiple queries.**\n\n**A3:** It is worth noting that in our architecture, the video encoding process and the question-answering process are disentangled. Our model only requires once forward computation to encode a video into memories, which are independent of specific questions. When encountering multiple questions for a video, we only need to calculate the similarity between multiple question indicators and the obtained time indicators to select different key clips. This process requires much fewer computations than the streaming video encoding and is efficient.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZxGDI5Pkdi", "reviewer": "Reviewer_sqPz", "summary": "This paper proposes VideoStreaming to understand arbitrary-length videos by streamingly encoding video tokens and adaptively selecting a constant number of them. Through careful designs, including a small LLM streaming encoder, prefix task, modified attention, memory propagation, and Gumbel-Topk token selection, the method successfully achieves superior performance on long video benchmarks, showcasing precise temporal comprehension for detailed question answering.", "questions": "1. Would it be possible to use the previous layers of Vicuna to replace the extra introduced (half) Phi-2, which has 1.3B parameters that cannot be ignored?\n\n2. Should L127-L129 be correct? StreamingLLM [a] has demonstrated the importance of initial tokens. I think it is your designed prefix task that restricts the streaming encoder from achieving this.\n\n3. The method still needs to segment clips. Why not directly operate on the frames with a moderate FPS (e.g., 2)? This should make the method more general. Or will this also incur a huge memory cost for the streaming encoder since it is still at the billion-level?\n\n4. The ablation in Table 8 is hard for me to understand. First, what is the performance if there is no memory and no selection? Second, the performance variance in MovieChat is so huge. Are these results from different trained models, or just the same trained model but different inference?\n\n[a] Efficient Streaming Language Models with Attention Sinks, ICLR 2024.", "limitations": "The limitations are discussed in the supplementary material: (1) simply uniformly sampling frames to form a set of short clips for memory-propagated streaming encoding rather than using adaptive segmentation; (2) the number of tokens may not be sufficient to represent clips with abundant visual content and intensive temporal dynamics. These are indeed limitations and are well discussed. However, as mentioned in the 'weakness' section, most limitations pertain to the experimental setting.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper has a strong motivation, aiming to solve the significant computational burden caused by the large number of tokens extracted from long videos. This is an important issue for LMM to understand long videos.\n\n2. The methodology is novel. First, the paper utilizes the causal LLM to satisfy the streaming and query-related vision summarization with explicit timestamp cues. Second, the paper designs a prefix task that naturally compresses the vision tokens during auto-regressive training. Lastly, through memory propagation and adaptive selection, the model successfully uses only the contextually related tokens to forward to the LLM, thus reducing the computational and memory cost due to the long kv cache.\n\n3. The paper is well written and easy to follow.", "weaknesses": "My main concerns are as follows: \n\nA. In common short/medium video length benchmarks, VideoStreaming could be unnecessary; \n\nB. For long video length benchmarks, the paper does not provide enough experiments; \n\nC. Promising experiments, like real-time interaction, are completely ignored in this paper.\n\nSpecifically,\n\nA.1. This paper actually uses much more data during training compared to LLaVA-Next-Video:\n\n- LLaVA-Next-Video-7B: 558K for MLP + 860K for MLP & LLM\n\n- VideoStreaming-8.3B: 790K for MLP + 763K for small LLM + 345K for all\n\nVideoStreaming is trained with 480K more data, which cannot be ignored. However, note that LLaVA-Next-Video-7B has slightly better results on VideoChat-GPT and even employs a much simpler pooling strategy to encode video tokens. Given that VideoChat-GPT's average duration is 1.81 minutes (see Table 1), we can infer that when the short video is less than 2 minutes, VideoStreaming may not offer advantages.\n\nA.2. For medium-length videos, such as EgoSchema (average 3 minutes), the paper seems to have missed the results of VideoChat2 mistral-7B, which shows an accuracy of 54.4%, much higher than the 44.1% reported in the paper. Although it is not entirely fair since VideoChat2 mistral-7B uses even more training data, its model employs a simple Q-former with 16 segments and 16 frames to handle the video. Therefore, I also suspect that VideoStreaming may not present advantages in medium-length videos. Here I have omitted the discussion of MovieChat-1K since its question setting does not really relate well to minute-long videos.\n\nB. Therefore, where VideoStreaming might truly excel is in handling very long videos. However, it is very disappointing that the paper does not present enough experiments on this. The ablations have been done on medium-length videos, and the authors do not provide any competitive baselines (with the same training data and procedure) in Table 7.\n\nC. In fact, VideoStreaming completely ignores a promising application, which is long-term, real-time streaming QA, such as GPT-4o (concept). The method can significantly reduce the kv cache consumption during long video streaming and maintain high-speed response even if a long sequence has been input.\n\nD. The demo shown in Figure 5(b) can be misleading. If you try the text-only Vicuna 13b (https://chat.lmsys.org/) and ask, \"What challenges does Philippe Petit face in his quest to walk between the Twin Towers?\" it can produce even better responses, such as \"getting access to the top of the Twin Towers,\" \"creating the perfect wire,\" \"practicing the walk,\" and \"keeping the walk a secret.\" This suggests that the question you provided may only prompt the LLM to recall its text pretraining data. I also doubt that the model can precisely select the relevant clips in a 2-hour movie, as that would be an extraordinary feat even for GPT-4o or Gemini-1.5-Pro.\n\nDespite the above weaknesses, I still acknowledge the novelty of the methodology. I will decide on my final rating after reading the rebuttal.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for recognizing our novelty, method and paper writing. Below we provide point-to-point responses to all the raised questions:\n\n> **Q1: The necessity of VideoStreaming on short-term videos.**\n\n**A1:** For short videos, sampling multiple frames and treating it as a multi-image understanding task works well. However, this is not suitable for long videos due to excessive number of frames. As we focus on long-form videos, the short video results are mainly to demonstrate our method also performs well in that setting. Since LLaVA-Next-Video is a concurrent work updated on April 30th, we did not include a comparison.\n\n> **Q2: The results on medium-length videos.**\n\n**A2:** The results of VideoChat2 Mistral-7B on EgoSchema is updated on May 23rd (in arxiv:2311.17005 v4), which is later than the submission. Technically, the accuracy of 54.4\\% is obtained by using 2M SFT data including ego-centric QA and more powerful LLM. Since it is difficult to obtain the same training data in a short period of time, we train VideoStreaming with Mistral-7B on our own data and compare the results on three benchmarks in Table 4 in rebuttal PDF.\n+ On EgoSchema, we achieve 48.1\\%. The gap to VideoChat2 is majorly due to the lack of ego-centric training data.\n+ On recent popular MLVU and VideoMME benchmarks consisting of medium and long videos, our model significantly outperforms VideoChat2 by 7.2 on MLVU multi-choice questions, 6.1 and 6.4 on VideoMME medium and long subset respectively.\n\n> **Q3: More experiments on long videos and explanation of Table 8.**\n\n**A3:** In rebuttal PDF, we provide more ablations on long videos in Table 6 and add explanations for different settings in Table.\n\nIn Table 6, we ablate the memory propagation, temporal selection, number of summarization tokens $P$ and number of selected clips $V$ on MovieNet-QA.\n+ For hour-long videos, the propagated memory is crucial for understanding from all aspects.\n+ The temporal selection significantly improves detailed plot analysis, and appropriately increasing the number of selected clips can further improve performance.\n\nFor the performance variance on MovieChat, the reasons are below.\n+ For global mode requiring long-term comprehension, the lack of memory will cause the LLM to have very limited temporal receptive fields and lead to dramatic performance drop. \n+ Besides, as shown in Figure 4 in manuscript, the beginning of the movie contains crucial cues for global understanding. Directly feeding the memories of final 4 clips without temporal selection into LLM also results in information loss.\n+ For breakpoint mode needing detailed understanding of specific moments, it is important to select the related clips otherwise the LLM cannot catch the necessary details. \n+ Further, the propagated memory across clips increases tolerance for imperfect temporal selection, as it preserves previous contexts. Without memory, there would be a strict requirement on temporal selection accuracy to avoid completely losing necessary details.\n\n> **Q4: The promising application like GPT-4o.**\n\n**A4:** Yes, VideoStreaming can disentangle video encoding and question-answering into two asynchronous processes for streaming QA with low time latency. We are working on the deployment to make it a more promising application. We did not report it in the paper due to the lack of suitable benchmarks for evaluation.\n\n> **Q5: About the example in Figure 5.**\n\n**A5:** Thanks for pointing out this. We find that some specialized words like 'Philippe Petit' and 'Twin Tower' can make LLM recall the text pretraining data. To solve this problem, we replace the words that may cause information leakage and formulate a new question as shown in Figure 1 in rebuttal PDF. The comparison between VideoStreaming and text-only LLM demonstrates that our model can comprehend the high-level expressions in the question and summarize the relative contents in the visual contexts.\n\n> **Q6: The grounding ability in Figure 5.**\n\n**A6:** We use a similarity-based strategy to select the clips related to specific questions, which relies on some semantic relations to do selection. Even with an implicit question expression, our model can correlate the scenes that contain the protagonist with the question as shown in Figure 1 in rebuttal PDF.\n\n> **Q7: Use previous layers of Vicuna to replace Phi.**\n\n**A7:** Using previous layers of Vicuna to replace half Phi will cause two problems.\n+ Our streaming encoder is trained with a prefix task to generate outputs that only reference memory tokens to build a robust memory. This requires first training the entire model for next token prediction, before using partial layers in later stages. Replacing Phi with Vicuna layers forces us to first train the full 7B model, significantly increasing computational cost.\n+ If we take the previous few layers of Vicuna and maintain the same parameters with Phi, we can only retain a very small number of layers. Table 5 in rebuttal PDF shows the performance drops when replacing Phi with previous Vicuna layers under the same parameters.\n\n> **Q8: The importance of initial tokens.**\n\n**A8:** Yes, you are correct. It is our prefix task that ensures the information of a sequence is distilled into the last few tokens. We will accordingly modify the presentation.\n\n> **Q9: Operate on the frames instead of clips.**\n\n**A9:** Yes, our model can be applied to the scenario by sampling frames at a moderate FPS. In Table 7 in rebuttal PDF, we show the comparison between clip-based sampling and frame-based sampling at 1 FPS, and present the average number of sampled frames on each dataset.\n+ The overall performance is close since the two sampling strategies lead to comparable number of sampled frames.\n+ On the longer MovieNet-QA videos, the frame-based sampling performs slightly better, as the clip-based approach limits the maximum number of clips, resulting in fewer total sampled frames for very long videos.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear reviewers,

We sincerely appreciate the constructive feedbacks provided by all the reviewers. The reviewers acknowledged some aspects of our work, including the motivation and novelty (Reviewer FUM4, sqPz), the significance of the research problem (Reviewer QjSL), the technical design (Reviewer FUM4, QjSL, sqPz), the thorough experiments (Reviewer QjSL), the efficiency in handling long videos (Reviewer 9Ye5), as well as the paper writing (Reviewer FUM4, sqPz).

Below we summarize some major points we have addressed in the rebuttal. More detailed responses are provided individually for each reviewer.

**More experimental comparisons:** We complement the experiments on IntentQA dataset (Reviewer FUM4), the comparison with LLaMA-VID on EgoSchema, Next-QA and MovieChat (Reviewer QjSL) and the comparison with VideoChat2 (Reviewer sqPz).

**More ablation studies and explanations:** We add the ablation studies on the instantiation of streaming encoder (Reviewer QjSL, sqPz), more ablation studies on hour-long MovieNet-QA dataset (Reviewer sqPz), and the explanations of different settings (Reviewer sqPz).

**More insights to the streaming encoder:** We provide further explanations on the design of summarization tokens (Reviewer QjSL), the necessity of language reasoning in memory formulation (Reviewer QjSL), as well as the influence of using different number of layers and different models as the streaming encoder (Reviewer QjSL, sqPz).

**Further explanations on the example of hour-long video QA:** We further explain the temporal grounding process (Reviewer QjSL, sqPz), and reformulate the question to avoid information leakage that may make LLM recall the text pretraining data for comparison (Reviewer sqPz).

**More discussions with realted works:** We clarify the differences between our propagated memory and the memory bank design in MemDPC (Reviewer QjSL), explain the significant zero-shot performance on EgoSchema compared to MC-ViT(Reviewer QjSL), and add discussion with LGDN on the clip selection process (Reviewer 9Ye5).

**Clarifications on some confusing points:** We provide clarifications on some confusing points, including the description of Gumbel-TopK technique (Reviewer FUM4), the progressive training (Reviewer QjSL), the efficiency of handling multiple queries per video (Reviewer 9Ye5), and the description in L127-L129 (Reviewer sqPz).

Due to space constraints, we have placed some of the added Figures and Tables in the uploaded author rebuttal PDF. In the detailed point-to-point responses below, we have specified the location of referred Figures and Tables. Please don’t hesitate to let us know if there are any additional clarifications or experiments that we can offer!

Yours Sincerely,

Authors

Meta-review: The authors proposed a model for streaming video understanding. Specifically, the authors break up a long video into multiple short clips, and encode each clip sequentially one-after-the-other, using a memory module to propagate information from one clip to the next. The authors also develop a memory selection method to extract the features from the memory which are most relevant to the caption. The authors show strong results on a number of video question-answering datasets.

During the rebuttal, the authors addressed the concerns from the reviewers, which led to most of them increasing their score. Ensure that all of these revisions are in the final camera-ready version of the paper. In the camera-ready version, please also discuss the relation to the memory model of [this recent published work](https://openaccess.thecvf.com/content/CVPR2024/papers/Zhou_Streaming_Dense_Video_Captioning_CVPR_2024_paper.pdf) on streaming video understanding.
Decision: Accept (poster)