Streaming Long Video Understanding with Large Language Models

Authors: rui  qian
xiaoyi  dong
pan  zhang
yuhang  zang
shuangrui  ding
dahua  lin
jiaqi  wang
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 10:27:00

Abstract: This paper presents VideoStreaming, an advanced vision-language large model (VLLM) for video understanding, that capably understands arbitrary-length video with a constant number of video tokens streamingly encoded and adaptively selected.
The challenge of video understanding in the vision language area mainly lies in the significant computational burden caused by the great number of tokens extracted from long videos. Previous works rely on sparse sampling or frame compression to reduce tokens. However, such approaches either disregard temporal information in a long time span or sacrifice spatial details, resulting in flawed compression. 
To address these limitations, our VideoStreaming has two core designs: Memory-Propagated Streaming Encoding and Adaptive Memory Selection. The Memory-Propagated Streaming Encoding architecture segments long videos into short clips and sequentially encodes each clip with a propagated memory. In each iteration, we utilize the encoded results of the preceding clip as historical memory, which is integrated with the current clip to distill a condensed representation that encapsulates the video content up to the current timestamp. This method not only incorporates long-term temporal dynamics into the streaming encoding process but also yields a fixed-length memory as a global representation for arbitrarily long videos. After the encoding process, the Adaptive Memory Selection strategy selects a constant number of question-related memories from all the historical memories, and feeds them into the LLM to generate informative responses. The question-related selection reduces redundancy within the memories, enabling efficient and precise video understanding. Meanwhile, the disentangled video extraction and reasoning design allows the LLM to answer different questions about a video by directly selecting corresponding memories, without the need to encode the whole video for each question. Through extensive experiments, our model achieves superior performance and higher efficiency on long video benchmarks, showcasing precise temporal comprehension for detailed question answering.

Main paper: 
 introduction: 
 The evolution of Large Language Models (LLMs) has significantly advanced artificial intelligence, encompassing text generation and reasoning in complex language environments [9, 74, 14, 61, 16, 69, 2, 70] . Later, the community extends LLMs to multi-modal domains, demonstrating promising results in captioning and question-answering tasks that integrate diverse visual signals [46, 40, 15, 59 ]. Yet, within the domain of video understanding, long video sequences pose a formidable challenge.Incorporating such long visual contents into LLMs requires a substantial number of tokens, which not only amplifies computational demands but also risks early contextual information loss [49] .Among the recent works on general video understanding with LLMs [48, 42, 44, 88, 50, 43, 66, 62] , a prevalent strategy is using sparse temporal sampling [44, 86] or spatio-temporal pooling [50, 48] to reduce tokens. Unfortunately, this paradigm explicitly loses substantial information in the long time span. To address this limitation, [43, 42, 88] develop frame-wise compression, with LLaMA-VID [43] as a typical example. It compresses each frame into only two tokens but overlooks the inter-frame temporal dynamics which are vital in compressing temporal redundancy within videos. Besides, its question-dependent compression pipeline limits the ability to produce a general representation that can handle diverse instructions. Another line of works employ memory banks [77, 7] to store history information [66, 22] . Whereas, these methods rely on explicit timestamps to recall the historical details, limiting the ability to generate comprehensive responses without specific time indicators.In this work, we propose VideoStreaming, a novel Memory-Propagated Streaming Encoding architecture with Adaptive Memory Selection to sequentially encode a long video into condensed memories and generate responses referring to relevant timestamps. The core idea behind the memory-propagated streaming encoding is to preserve representative spatial cues and temporal dynamics while reducing temporal redundancy in videos. To achieve this goal, we segment the long video into multiple short clips and sequentially encode each clip. When encoding each clip, we first refer to the encoded results of its preceding clip as historical memory, then concatenate it with the current clip features and feed them into a small decoder-only language model [20] . Due to its autoregressive nature, the information of the sequence naturally accumulates to the last few tokens [38, 34] . Consequently, we take these last few tokens as an updated memory that encapsulates the video information up to the current timestamp. Through this streaming encoding, we explicitly take long-term temporal relations into consideration and maintain a fixed-length memory to represent an arbitrarily long video.However, this fixed-length memory inevitably loses detailed information, especially in early contexts. To address this problem, we store the historical memories of all clips and select a constant number of subsets that are closely related to the question. To accomplish this, when streaming encoding each clip, we additionally append a summary token at the end of the sequence as a clip indicator that summarizes the clip contents within one token. Then, given a specific question, we concatenate the condensed memory from the final iteration with the question and pass it through the same small language model used in streaming encoding. We take the final token as the question indicator and calculate its similarity with all historical clip indicators, the clip indicator with higher similarity means its corresponding memory is more related to the question. Finally, we feed the adaptively selected memories into the LLM for detailed question answering.In practice, we realize our VideoStreaming with a carefully designed two-stage progressive training process and long-video data construction strategy. In the first stage, we empower a small language model with the single-clip encoding capability by a specialized prefix task. In the second stage, it serves as the streaming encoder and we jointly train it with the LLM for long video understanding. Due to the lack of long video QA data, we manually constructed a set of long video QA pairs in two ways. On the one hand, we concatenate short videos from existing datasets [76, 81] into longer ones, where the original questions correspond to different segments. On the other hand, we curate a subset of Panda-70M [11] which includes captions for segmented clips as well as the original long videos, and use this to create multi-round long video QA pairs with explicit timestamps. These long video QA data not only optimize the responses from the LLM but also guide accurate memory selection.In summary, our contributions are as follows: (1) We analyzed the challenge of long video understanding in the vision language area, and pointed out that the problem of current methods lies in the inefficient video encoding. (2) In response to the challenges, we propose two efficient designs: Memory-Propagated Streaming Encoding and Adaptive Memory Selection, which result in our advanced video understanding model VideoStreaming. (3) The extensive experiments demonstrate that our model achieves precise temporal grounding with respect to specific questions, attains superior performance, and exhibits higher inference efficiency on long video benchmarks. 
related work: 
 Large Language Models (LLMs) have revolutionized natural language processing. Early works establish encoder-decoder models with masked language modeling [16, 61] , while later decoder-only Long Video Understanding is a challenging task in computer vision. The most prevalent strategy is to maintain a memory bank to store history information in long videos [77, 7, 78, 54, 12, 71, 21] . Typically, MemDPC [21] formulates a memory bank shared across the whole dataset as an external knowledge reference for future prediction, but it is not explicitly designed for long-term understanding. To facilitate long video content analysis and ensure computation efficiency, MC-ViT [7] , MovieChat [66] and StreamingCaption [89] rely on handcrafted rules like clustering to compress the history into a finite-length memory and iteratively update it in video streams. A potential drawback is that the memory cannot be optimized through the extensive video caption data. It is potentially more promising to learn comprehensive memory consolidation in a data-driven manner. More recently, [32, 27, 87] use language as a bridge for long-term video understanding. They first divide a long video into short clips, generate textual descriptions for each clip, and then employ an LLM to aggregate the short captions for long video analysis. However, this architecture cannot be trained end-to-end, and the long video understanding quality depends on the short clip captions. In contrast, we employ a trainable small language model to iteratively encode short clips into compact memories, which can be jointly optimized with the subsequent LLM for long video understanding. 
videostreaming: 
 In this section, we introduce VideoStreaming, a streaming long video understanding framework with LLM. As illustrated in Fig. 1a , given a long video input, VideoStreaming segments it into multiple short clips and iteratively encodes each clip into compact historical memory. To enhance the reasoning ability to specific questions, we design an adaptive memory selection strategy to select a subset of relevant memories and feed them into an LLM to produce detailed responses.To effectively distill the information within a sequence into a compact set of tokens, we take inspiration from recent advanced decoder-only language models [2, 70, 13, 5, 29] and employ a comparatively small language model, Phi-2 [20] , for efficient encoding. Due to the causal attention and autoregressive nature, it is intuitive to utilize a language model to aggregate the sequence information onto the last few tokens [38, 34] , which naturally serve as a compact representation that provides a high-level summary of the input sequence.Mathematically, given a T -frame video clip, we first use a pre-trained CLIP ViT-L [59] to extract frame-wise features and concatenate every four spatially adjacent visual tokens along channel dimension to reduce the number of tokens by 75%. The resulting clip features are denoted as F ∈ R T N ×C , where N denotes the per-frame spatial token number, and C is the channel dimension. To produce the condensed representations, we initialize a set of summarization tokens S ∈ R T P ×C by adaptively pooling each frame into P tokens, where P ≪ N . Intuitively, S can be regarded as a Assistant: The man raises his left arm in the middle of the video to grab the baby's hand. 1a shows an overview of VideoStreaming, where we segment a long video into short clips and iteratively encode each clip into compact memories. Then, according to specific questions, we select a constant number of subsets of relevant memories as input to an LLM to produce responses. The ✓ and ✗ respectively denote selected and unselected memories. Fig. 1b illustrates the detailed process of each streaming encoding iteration. We encode current clip features with reference to specific timestamps and historical memory from the preceding clip into a condensed representation. coarse encapsulation of the given clip, making it well-suited to serve as the summarization tokens for consolidating the clip information. To this end, we concatenate F with S and feed them into the encoder g(•), which consists of an MLP projector and a language model Phi-2. We utilize the output of the last T × P tokens as the condensed representation of the given clip:EQUATIONEQUATIONwhere • denotes concatenation operation, D is the channel dimension of Phi-2.To reinforce the visual consolidation ability, we design a prefix task to train the encoder on visual captioning and question-answering tasks. In particular, to guarantee that the clip information is distilled into the summarization tokens, we enforce the language model to generate the response only with reference to these few tokens. To achieve this goal, a straightforward way is to modify the attention mask in each Transformer decoder layer. As depicted in Fig. 2 , we take a sequence covering T N clip feature tokens, T P summarization tokens, and T T text response tokens as an example.Based on the standard causal attribute, the binary attention mask M is modified as shown in Figure 3 : with the modified attention mask, the T T text tokens can only get video-related information from the T P summarization tokens to predict the next token. This encourages the summarization tokens to extract more video information from previous T N video clip tokens, ie. learns better video encoding.𝑭 ∈ ℝ !"×$ 𝑺 ∈ ℝ !%×$ Text response 𝑯 ∈ ℝ !%×$ Small Language Model Text response Prefix taskTill this point, we have obtained an encoder capable of distilling short video clips into condensed representations. The next step is to comprehensively consider the long-term temporal relations within the complete videos, leveraging the historical information from previous clips to facilitate the encoding of subsequent segments as depicted in Fig. 1b .To accomplish this objective, we divide a long video into K clips, each containing T frames, and propose a memory-propagated streaming encoding mechanism to iteratively encode each clip in sequence. In each iteration, we employ the encoded results from the last iteration as historical memory and integrate them with current clip features to produce an updated memory for subsequent encoding. Specifically, given the k-th clip, we denote the current clip features as F k ∈ R T N ×C , the summarization tokens as S k ∈ R T P ×C , and an additional global token as Ŝk ∈ R 1×C . This global token, initialized by global average pooling on the clip features F k , is expected to summarize the entire clip contents and serve as a clip indicator for memory selection in the next subsection.To enrich the temporal contexts, we refer to the encoded representations from the previous clip H k-1 ∈ R T P ×D to provide historical information. Then we jointly feed them into the streaming encoder to produce the condensed representation H k ∈ R T P ×D and the clip indicator Ĥk ∈ R 1×D of the k-th clip:H k , Ĥk = g([H k-1 • F k • S k • Ŝk ]).(2) Note that for the first clip encoding, the historical memory is not used. Through this streaming encoding process, H k not only encompasses the current clip information but encapsulates the overall video content up to the k-th clip. To this end, we manage to maintain a fixed length of memory to represent arbitrarily long videos.Discussion. In this architecture, we use a language model for video encoding, which has the unique advantage that we can flexibly provide the encoder with diverse prompts to guide the encoding process. Hence, the summarization tokens capture not only the core content but also additional contextual information. Typically, the explicit timestamp is an important cue in videos [62] . As shown in Fig. 1b , we incorporate a text prompt indicating the specific timestamps of each clip and historical memory to enhance temporal awareness. Besides, this prompt-based approach also allows the user to tailor the condensed output to better suit the needs of downstream tasks, going beyond a purely extractive summarization.Another noteworthy point is that in the language model, the feature space of the final decoder layer is designed for the next token prediction, which may not perfectly align with the objective of producing condensed video representations. Considering that we modify the attention masks in each decoder layer to encourage information consolidation, this allows us to leverage the intermediate outputs from partial attention layers as the encoded results. Similar to the techniques in vision domain [82, 46, 17] , this strategy potentially enables the model to capture a richer set of semantic and contextual features as the condensed representations, bridging the gap between the language model's original training objective and the requirements for video encoding.Through the streaming video encoding, it is feasible to use the encoded results from the final iteration, i.e., H K , as a compact global memory that concludes the entire video. However, this fixed-length memory inevitably loses details, especially the information from early segments. Hence, this global memory alone is insufficient for comprehensive long video understanding.To address this limitation, we make use of the encoded results of all historical clips of the input video, i.e., H = {H 1 , H 2 , ..., H K }. Given a specific question or instruction, we first generate an adaptive indicator that summarizes relevant video content for that particular instruction. We accomplish this by reusing the language model in the streaming encoder, where we concatenate the global memory from the final iteration, H K , and the instruction texts, then pass the sequence into the model. We employ the output of the final token as the instruction indicator, denoted as ĤQ ∈ R 1×D . Thereafter, we calculate the cosine similarity between this instruction indicator and all historical clip indicators { Ĥ1 , Ĥ2 , ..., ĤK } ∈ R K×D and obtain the similarity distribution s ∈ R K . To achieve a differentiable discrete selection, we adopt Gumbel-Topk technique [36] to produce a binary index I that activates a subset of V out of K positions with the highest similarities:I = Gumbel-Topk(s, V ) ∈ {0, 1} K .(3) Based on I, we select the corresponding encoded results from H to formulate a subset of memories that are related to the instruction:EQUATIONwhere I k denotes the selected indexes. We concatenate the selected memories Ĥ in temporal order, resulting in a sequence consisting of V × T × P tokens. Then, we feed the sequence with instruction texts into an LLM for comprehensive reasoning.Table 1 : The statistics of the average video duration time of each evaluation dataset.Next-QA [79] 42.23 sec Next-GQA [80] 39.60 sec VideoChatGPT [50] 1.81 min EgoSchema [51] 3.00 min MovieChat-1K [66] 7.66 min MovieNet-QA [68] 108.26 min Our adaptive memory selection allows the model to dynamically access historical memories relevant to specific instructions, which mitigates the information loss inherent in the streaming encoding process. By drawing upon fine-grained details across the full video duration, the LLM can provide detailed and informative responses, while preserving high computational efficiency.To train VideoStreaming, we design a progressive two-stage paradigm. First, we train single clip encoding on image and short video understanding tasks. Next, we train memory-propagated streaming encoding and adaptive memory selection as well as the LLM for long video understanding. Single Clip Training. In this stage, both image-and video-text pairs are used to train the encoder to handle general visual signals. Following [44, 50, 88, 43] , we employ 790K image and short video caption data [64, 6] to train the MLP projector for modality alignment. After that, we employ 763K image and video instruction data from [46, 50, 45] to finetune the small language model. For video input, we uniformly sample T = 16 frames with spatial resolution 224 × 224 and use a frozen CLIP ViT-L/14 [59] to extract frame-wise features. After adjacent token merging, we obtain 16×64 = 1024 tokens as the clip feature representation. Then, the encoder, a two-layer MLP and a small language model Phi-2 2.7B [20] , distills each frame into P = 4 tokens, resulting in 16 × 4 = 64 tokens as the condensed representation with a compression ratio of 16 : 1. For image-text pairs, we regard the images as single-frame clips and encode each into 4 tokens. We use standard next token prediction to consolidate visual contents into compact summarization tokens as illustrated in Fig. 2 .Streaming Long Video Training. In the second stage, we use long video QA pairs to finetune the whole architecture, including ViT, the streaming encoder, and the LLM, as shown in Fig. 1a . The long video QA data encompasses three parts. (1) We adopt 25K movie QA pairs from [43, 24, 66] . (2) We curate a subset from Panda-70M [11] , which provides the original long videos and the captions of segmented clips. Based on this subset, we create 300K multi-round long video QA pairs with explicit timestamps. (3) We synthesize 20K long videos by concatenating short videos from existing QA datasets [81, 76] , and the original QA pairs correspond to different segments in the synthesized long videos. For each video, we extract 16-frame clips at 1 FPS, and the number of clips varies with the video duration. In streaming encoding, we employ the intermediate outputs from the first 16 layers of Phi-2 as the condensed memories. Finally, we select V = 4 most relevant timestamps and feed the selected memories of V × T × P = 256 tokens into the LLM, Vicuna-7B [13] , for long video reasoning. Since our curated long video data could provide pseudo temporal grounding labels of specific questions, we utilize 30K QA pairs to warm up memory selection via a KL divergence loss. Subsequently, we use the rest 315K QA pairs to optimize the responses from the LLM and guide memory selection in a weakly-supervised manner. More training details are included in Appendix A. 
datasets: 
 We evaluate our model on long video QA datasets and present the statistics on the temporal duration of individual datasets in Table . 1. Among them, Next-QA [79] , Next-GQA [80] and VideoChatGPT [50] encompass minute-long videos with thousands of frames. EgoSchema [51] contains over 5K threeminute videos with multiple-choice questions. Each question has a long temporal certificate, requiring more than 100 seconds within a video to produce a correct answer. MovieChat-1K [66] and MovieNet-QA [68] consist of around ten-minute-long or even hour-long movies, posing significant challenges for the model to comprehend the visual contents across such long time spans. In this section, we present the results of our 8.3B model (half of Phi-2 2.7B in streaming encoder and Vicuna-7B as the LLM). We omit the comparisons to proprietary LLMs.VideoChatGPT. Table 2 presents the results on VideoChatGPT [50] in terms of Correctness of Information (CI), Detailed Orientation (DO), Contextual Understanding (CU), Temporal Understanding (TU) and Consistency (CO). Our model outperforms LLM-based video understanding methods on all five metrics, with a significant advantage in temporal understanding. It can be attributed to the memory-propagated streaming encoding architecture that explicitly captures temporal dynamics. EgoSchema. In Table 3 , we report the zero-shot performance on the fullset test split of EgoSchema [51] . MC-ViT [7] consolidates a long-term memory to memorize long contexts but requires finetuning on related dataset [19] . LLM-based methods [87, 32, 72] curate answers from the captions of segmented video clips. However, these short-term captions cannot be optimized end-to-end and inevitably lose some detailed information. In contrast, we use a trainable streaming encoder to produce memory embeddings in long videos and feed them into an LLM to generate responses. Our model outperforms all zero-shot methods and is comparable to the finetuned MC-ViT, demonstrating the effectiveness of our streaming architecture for long-term temporal modeling. Next-QA. In Table 4 , we perform zero-shot evaluation on the validation split of Next-QA [79] covering 5K multiple-choice questions. We respectively report the accuracy on Causal (C), Temporal (T) and Descriptive (D) subsets. Our method consistently surpasses all zero-shot counterparts. Typically, compared to LangRepo [32] with Mixtral-8×7B [30] , our 8.3B model improves the causal, temporal, and descriptive accuracy by 0.7%, 10.8%, 9.0% with considerably fewer model parameters.Next-GQA. Besides the evaluation of the generated responses, we also assess the temporal grounding ability on Next-GQA [80] . We calculate the Intersection of Prediction (IoP) and Intersection of Union (IoU), and use Acc@GQA to measure the accuracy of the correctly grounded predictions. According to the comparisons in Table 5 , our simple similarity score based selection achieves the highest IoP and comparable IoU to SeViLA [86] with a specialized grounding module. Moreover, the highest Acc@GQA demonstrates the comprehensive capacity for grounding and high-level understanding. MovieChat-1K. Table 6 shows the results on MovieChat-1K [66] , including a global mode for overall long-term understanding and a breakpoint mode for detailed analysis of specific moments. In breakpoint mode, [66, 67] manually extract segments according to the timestamps in questions, while our model adaptively selects the related historical memories. Fig. 4 reveals that our selected timestamps are close to the ground-truths, and the higher breakpoint accuracy validates our adaptive selection effectively gathers the desired information from long contexts. Meanwhile, we reach significantly superior results in global mode, with the model's selection concentrated at the beginning and ending parts. On the one hand, the beginning of a movie often contains hints of global information while the middle comprises redundant details. On the other hand, the condensed memories near the end of the video encapsulate the entire video, making them quite suitable for global understanding. MovieNet-QA. Finally, we show the results on MovieNet-QA [68] consisting of 100 hour-long movies. Inspired by [68] , we use GPT-3.5 to produce scores in range 0-5 to evaluate the performance in overview, plot, and temporal understanding in Table 7 . Specifically, LLaMA-VID [43] compresses each frame into two tokens, which are then combined with movie subtitles as input to an LLM. MovieLLM [68] further incorporates more generated data in training. These approaches largely rely on the texts for movie understanding, and only using visual frames leads to dramatic performance drop. Moreover, its frame-wise compression is dependent on specific questions. The model has to reprocess the entire movie to extract visual features for different questions, resulting in a high inference latency of over 10 seconds per question. Conversely, our architecture requires only once streaming encoding to obtain a general condensed representation and adaptively selects significantly fewer tokens as input to LLM to answer specific questions. Therefore, we achieve a higher inference speed of 5.32 seconds per question and attain promising movie understanding without using subtitles.Qualitative Results. We also present qualitative examples in Fig. 5 . Typically, in Fig. 5a , our model accurately captures the detailed descriptions in the question, and precisely selects the relevant segments that contain the corresponding character. Moreover, in Fig. 5b , given a two-hour long movie and a high-level question on the movie plot, without relying on subtitles, VideoStreaming can comprehend the intent of the question and select relevant scenes from the lengthy video. In particular, the model selects the scenes of tightrope walk, team disputes, and equipment setup, clearly illustrating the protagonist's challenges, thereby contributing to a comprehensive answer generation.We first explore four different settings on the memory and selection design of our model in Table 8 .The details are as follows: Historical Memory. In terms of the propagated memory in the streaming encoding process, i.e., H k-1 in Eq 2, we report the fullset accuracy on EgoSchema [51] as well as global and breakpoint accuracy on MovieChat-1K [66] in Table 8 . Typically, the historical memory significantly improves global understanding. This verifies our intuition that leveraging historical memory enables the model to produce a global representation that summarizes the entire video. Meanwhile, since we select a small portion of the encoded results from the long video as input to LLM, the propagated memory across clips increases tolerance for imperfect temporal selection, as it preserves previous contexts. Without memory, there would be a strict requirement on temporal selection accuracy to avoid completely losing necessary details, thus degrading the performance. Memory Selection. We also validate the effects of our memory selection strategy. Comparing the results in Table . 8 with respect to temporal selection, we have two observations. First, for breakpoint mode needing detailed understanding of specific moments, the lack of temporal selection leads to dramatic performance drop. It is crucial to select the related clips otherwise the LLM cannot catch the necessary details. Second, the historical memories in streaming encoding process enable the encoded results from the final iterations to provide coarse summarization of the entire video. Whereas, as shown in Fig. 4 , the beginning of the movie contains crucial cues for global understanding. Directly feeding the memories of final clips without temporal selection into LLM still results in information loss. These phenomena demonstrate the necessity of our adaptive selection for gathering detailed information over the long time span, which facilitates more accurate and informative responses. Sampling Strategy. In default, we use clip-based sampling to segment a video into clips each consisting of a fixed number of frames. Alternatively, it is also feasible to directly sample at a moderate FPS, e.g., 1 FPS, and streamingly process the frames. Table 9 showcases the comparison between clip-based sampling and frame-based sampling at 1 FPS, and presents the average number of sampled frames on each dataset. The overall performance is close, which verifies that our model can well generailze to different scenarios. And on the longer MovieNet-QA videos, the frame-based sampling performs slightly better, as the clip-based approach limits the maximum number of clips, resulting in fewer total sampled frames for very long videos. Language Reasoning in Streaming Encoding. Besides, we ablate the necessity of using a language model for memory consolidation. We compare our streaming encoder instantiated with partial layers of Phi-2.7B [20] with a conventional ViT-based approach, MC-ViT [7] . We compare the results on EgoSchema fullset [51] and Next-QA [79] , Global and Breakpoint Accuracy on MovieChat-1K [66] , as well as the number of encoder layers and parameters in Table 10 . It is clear that using language reasoning to extract video memories exceeds the vision-based strategy even with fewer parameters. The reasons are twofold. On the one hand, MC-ViT updates memory according to handcrafted rules like clustering. In contrast, using language model as the streaming encoder allows us to leverage extensive video caption/QA data to guide memory consolidation in an end-to-end manner. This datadriven strategy results in more comprehensive memories that facilitate general video understanding. On the other hand, since the memory is fed into a subsequent LLM for reasoning, it is easier to align the memory generated by a language model with the input space of LLM, thus improving performance.Streaming Encoder Architecture. Additionally, we also explore two alternative language models as the streaming encoder. (1) A comparatively a small language model, Phi-2.7B [20] , with different partial layers. (2) The previous layers of the large language model, Vicuna-7B [13] . Interestingly, using comparatively fewer layers of the language model leads to better results. We conjecture this is because the language model is originally trained for next token prediction. Its feature space of the final Transformer decoder layer might not align with the objective of visual content condensation. Similar to [82, 46, 17] , the shallower layers might produce feature embeddings that encode richer information and serve as more comprehensive condensed video representations. Besides, under the same parameters, the smaller language model retains more layers and consistently outperforms the previous layers of Vicuna. Meanwhile, the prefix task for training the streaming encoder requires first training the entire model for next token prediction, before using partial layers in later stages. Therefore, using Phi as the streaming encoder brings another advantage in reduced computation. More ablation studies on temporal grounding supervision, the number of summarization tokens and selected timestamps, the time prompts, and the similarity measurement are included in Appendix D. 
conclusion: 
 In this paper, we introduce a novel approach to tackle the complexities of long video understanding with large language models (LLMs). Our proposed memory-propagated streaming encoding architecture segments long videos into short clips and iteratively encodes each clip in sequence. By leveraging historical memory from preceding clips, we incorporate temporal dynamics into the encoding process and produce a fixed-length memory to encapsulate arbitrarily long videos. To further augment the detailed information for handling specific questions, we develop adaptive memory selection that selects relevant timestamps based on given instructions. This approach ensures that the most pertinent historical memories are utilized for question answering, thereby facilitating detailed and informative responses. Our model achieves superior performance with substantially fewer tokens and higher efficiency on extensive long video benchmarks. We demonstrate that memories from the streaming encoding significantly enhance global video understanding, while adaptive selection results in accurate temporal grounding with respect to specific questions. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The potential impacts are discussed in Section 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes]Justification: We will require users to adhere to specific usage guidelines to access the model and datasets, ensuring that they are used responsibly and to mitigate the risk of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have properly credited the creators or original owners of assets used in the paper and we use the license CC-BY 4.0.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: One potential limitation is that we simply uniformly sample frames to form a set of short clips for memory-propagated streaming encoding. However, in a long video, different segments possess different amounts of information. The uniform sampling may result in using redundant tokens for clips with bland content. Meanwhile, the number of tokens used to represent clips with abundant visual contents and intensive temporal dynamics may be insufficient, leading to information loss. To address this limitation, we plan to explore adaptive segmentation techniques that dynamically adjust the segmented clip lengths based on the complexity and content of the video.Our proposed VideoStreaming, a streaming long video understanding architecture with large language models has various potential impacts for society. On the positive aspect, VideoStreaming contributes to improved intelligent video understanding, especially for long videos. This could be beneficial education, entertainment, and information retrieval, where users often need to navigate and understand complex video materials. Besides, our technique could lead to advancements in multimedia analytics with applications in areas like video surveillance, market research, and content personalization.On the negative aspect, the ability to efficiently process and retrieve information from long videos raises potential privacy and security concerns. If misused, this technology could be employed for unauthorized surveillance, personal monitoring, or other unethical purposes that infringe on individual privacy. In addition, the enhanced video understanding capabilities might be exploited for the creation of manipulated or misleading video content, leading to the spread of misinformation and the potential for social manipulation.In conclusion, despite that VideoStreaming presents advancement in long video comprehensive, its development should be accompanied by careful consideration of ethical and societal implications.We use CLIP ViT-L/14 [59] to extract frame-wise features with input resolution 224×224, resulting in 256 tokens per frame. Then, we concatenate every four spatially adjacent visual tokens along channel dimension, representing each frame with 64 tokens with channel dimension 4096. The streaming encoder consists of a two-layer MLP projector (channel dimension 4096-2560-2560) with GELU activation [23] and a language model Phi-2 2.7B [20] . In the first training stage, we initially freeze Phi-2, and only tune the MLP projector on 790K caption pairs, including 558K image caption data from CC3M [64] and 232K short video caption data from WebVid 2.5M [6] . Following LLaVA [46, 45] , we use AdamW optimizer [47] with global batchsize 256, initial learning rate 1 × 10 -3 with cosine decay to train 1 epoch for modality alignment. Subsequently, we jointly train Phi-2 and the MLP projector on 763K QA pairs, including 625K image QA pairs [18, 26, 33, 37, 46, 52, 53, 63, 65] , 40K text conversations [1] and 98K video QA pairs [10] , with global batchsize 128, initial learning rate 2 × 10 -5 with cosine decay.In the second stage, we jointly train ViT, the streaming encoder and the LLM on long video data. In the memory-propagated streaming encoding process, we insert a brief prompt to indicate the explicit timestamps of the historical memory and the input clip formulated as This contains a history of {start} to {end} seconds, and a clip sampled in {start} to {end} seconds.. We adopt the output of the first 16 layers out of the 32 layers of Phi-2 as the condensed representation. Then, we adaptively select 4 most relevant timestamps and feed the associated 256 memory tokens into a two-layer MLP projector with channel dimension 2560-4096-4096 and an LLM, Vicuna-7B [13] to generate the final responses. We jointly train the whole architecture, including Vicuna, Phi-2, MLP projectors and ViT encoder, on long video QA data with global batchsize 128, initial learning rate 2 × 10 -5 with cosine decay. In default, we first use 20K synthesized long videos and sample 10K QA pairs curated from Panda-70M with pseudeo temporal grounding labels to train memory selection as warm-up. The learning objectives contain a standard next token prediction loss and a supervised KL divergence loss that aligns the distribution of the predicted memory selection results and the pseudo temporal labels. Next, based on the warm-up model, we further train on the rest 295 long video QA pairs only with Data provided by Panda-70M: [1, 12] : The speaker is a female news reporter who is sitting in front of a studio camera. [15, 23] : A group of people standing in line outside of a building. [25, 28] : Two children standing next to each other on a tv screen. next token prediction loss. The whole training is conducted on 32 A100 (80G) GPUs for around 2.5 days.In addition to the existing 25K long video QA pairs on movies [66, 43] , we create more QA data from two aspects. First, we leverage the existing short video QA dataset [76, 81] and synthesize short videos into minute-long videos with average duration of one minute. The original questions of each short video coarsely correspond to a temporal segment in the synthesized long video. We use this correspondence as noisy labels to supervise the memory selection. Second, recent Panda-70M [11] segments long videos into short clips and produces captions for each clips. This dataset provides the original long videos, the captions of segmented clips as well as the segmentation timestamps. Based on these cues, we produce multi-round QA conversations. Below we show an example in Fig. 6 . The produced time-sensitive QA pairs are crucial to enhance the temporal awareness and guide precise memory selection in long videos.Quantitative Results on IntentQA. IntentQA [41] is a long-form video understanding dataset consisting of 4.3K videos with 16K multiple-choice questions, which are classified into three types, why, how and before/after. In Table 11 , we report the zero-shot performance on IntentQA test set. Our method presents a dominant advantage in temporal understanding. And the overall performance is significantly superior to recent works. Analysis on the Selected Clips. To verify whether the selected clips are redundant, we provide statistics on the feature similarity and the temporal distribution of the selected clips in Fig. 7 . For feature similarity, we calculate the cosine similarity of the time indicators of the selected clips. We divide the x-axis from 0 to 1 into 20 bins, and it shows the distribution of feature similarity. The average cosine similarity is 0.68. For temporal distance, we calculate the time intervals between the selected clips (represented as the ratio of the total video duration) and visualize the distribution of these time distances. The average distance is around 35% of the total video length. The statistics on the feature similarity and temporal distance indicate that the selected clips are not redundant.We provide more ablation studies on the use of temporal grounding supervision, the number of summarization tokens and selected timestamps, the effects of time prompts in the memory-propagated streaming encoding process, and the similarity measurement used in memory selection.Temporal Grounding Supervision. First, we present the studies on the use of temporal grounding supervision. As mentioned in Section 3.4, we employ around one-tenth of long video QA pairs to provide pseudo temporal labels. We compare four training strategies: (1) Fully weakly-supervised manner without any pseudo labels. (2) Using pseudo labels to train a warm-up model, then expanding to large-scale QA pairs. (3) Mixing all long video QA data, where the model uniformly receives temporal supervision in training. ( 4) Training on mixed data after warm-up initialization. The results on EgoSchema [51] and Next-GQA [80] in Table 12 indicate three key points: First, warm-up training contributes to more powerful grounding ability. The sparse temporal label supervision in mixed mode is overcome by the powerful initialization from warm-up training, which can generalize to large-scale data. Second, reusing the temporal labels after warm-up offers no additional benefits, so we adopt warm-up as the default setting. Third, without using temporal labels, the grounding performance drops, but the QA accuracy remains stable. Fig. 8 reveals that compared to those trained with temporal labels, the weakly-supervised model selects relatively later segments that preserve previous contexts with the help of historical memory, thus maintaining comparable QA capacity.The Number of Summarization Tokens and Selected Timestamps. We compare using different number of summarization tokens and selected timestamps, i.e., P in Eq. 1 and V in Eq. 3. WeTable 13 : Ablatoin study on the number of summarization tokens and selected timestamps. We report the results on EgoSchema [51] and Next-GQA [80] . P V Tokens Fullset Acc@GQA compare the performance as well as the number of tokens input to LLM in Table 13 . We conclude three observations. First, too few summarization tokens, e.g., P = 1, leads to substantial performance drop, since it condenses a 16-frame into only 16 tokens with significant information loss in spatial contexts. Such information loss cannot be compensated by selecting more temporal segments. Second, the performance saturates when improving P from 4 to 16. This is because the existing video benchmarks [51, 80] do not place high demands on spatial detail understanding. It is sufficient to represent each frame with 4 tokens on average. Third, increasing the number of selected timestamps only results in minor improvements, which is not proportional to the increased number of tokens. This can be attributed to the historical memory used in the streaming encoding process. The utilization of historical memory enables the condensed representation of each clip to encompass the information in preceding clips, which enlarges the temporal receptive field. Hence, increasing the number of selected timestamps does not proportionally increase the temporal receptive field, resulting in slight performance improvements.Time Prompts. We explore three different formulations of the time prompts used in memorypropagated streaming encoding: (1) Only with the timestamps of the current clip, e.g., This clip is sampled in {start} to {end} seconds. (2) Only with the timestamps of the historical memory, e.g., This contains a history of {start} to {end} seconds. (3) Simultaneously with the timestamps of the historical memory and the current clip, e.g., This contains a history of {start} to {end} seconds, and a clip sampled in {start} to {end} seconds. We report the results of different time prompts in Table 14 .It is obvious that the lack of time prompts leads to substantial performance drop in the MovieNet-1K breakpoint mode accuracy, which requires detailed analysis of specific moments. The reason is that the breakpoint mode requires the model to answer questions at specified timestamps, the time prompts provide the model with necessary information in adaptive selection. Meanwhile, incorporating the timestamps of historical memories results in more significant improvements in global understanding. Overall, jointly leveraging the memory and clip timestamps contributes to the best results. Similarity Measurement. Besides, we present the study on the similarity measurement used in adaptive memory selection. We compare the default cosine similarity against simple dot product without normalization in Table 15 . Empirically, we observe that dot product could result in numerical instability, leading to overflow in training. Consequently, the calculated similarity score cannot reflect the correlation between the instruction and different segments and results in poor results on questions that require accurate temporal grounding, e.g., the Acc@GQA metric on Next-GQA [80] and the breakpoint mode accuracy on MovieChat-1K [66] .Analysis on Long Video Benchmark. Finally, we show the ablation studies of different settings particularly on the hour-long video benchmak, MovieNet-QA in Table 16 . It is clear that for hour-long videos, the propagated memory is crucial for understanding the overview contents, detailed plots as well as temporal relations. Meanwhile, the temporal selection significantly improves detailed plot analysis, and appropriately increasing the number of selected clips can further improve performance.NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. Justification: The information on the computer resources is provided in Appendix A.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: The research adheres to the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 


Decision: The authors proposed a model for streaming video understanding. Specifically, the authors break up a long video into multiple short clips, and encode each clip sequentially one-after-the-other, using a memory module to propagate information from one clip to the next. The authors also develop a memory selection method to extract the features from the memory which are most relevant to the caption. The authors show strong results on a number of video question-answering datasets.

During the rebuttal, the authors addressed the concerns from the reviewers, which led to most of them increasing their score. Ensure that all of these revisions are in the final camera-ready version of the paper. In the camera-ready version, please also discuss the relation to the memory model of [this recent published work](https://openaccess.thecvf.com/content/CVPR2024/papers/Zhou_Streaming_Dense_Video_Captioning_CVPR_2024_paper.pdf) on streaming video understanding.

Reviews: review: paper_summary: paper_summary: In this paper, a vision-language model, named VideoStreaming, is proposed. 
It adopts the memory based architecture to understand long video. Specifically, it segments a long video into multiple short clips and encodes each clip sequentially. During this encoding process, it exploits the memory feature to propagate the information from previous clips. Also, it develops the memory selection strategy to find out question-related memory among all encoded memories.
Extensive experimental results on various VQA datasets show that the proposed algorithm achieves better performances than exisiting methods.
strength_weakness: strength_weakness: I could not find the critical flaws in this work. 
Even though the motivation is not very new in long video understanding, the proposed memory-based framework is reasonable and well-designed.

- It would be better to provide the comparsion on IntentQA since it is also widely used for LVQA.
- In L201, the authors said 'we develop a variant of Gumbel-Softmax, denoted as Gumbel-TopK(),...' However, there is no detailed description on Gumbel-TopK in the paper and Gumbel-TopK trick is already widely used. Thus, I would recommend using other expression, instead of 'develop.' Otherwise, it would be better to explain Gumbel Top-k in detail.
questions: questions: In overall, I think that the motivation of the paper is reasonable. Also, the proposed memory propagating encoding framework and memory selection method have some contribution to the field. Therefore, I'm positive about this work. Please find my concerns in the weakness section.
limitations: limitations: Yes, L574-581
paper_summary: paper_summary: This paper presents an advanced vision-language model designed to handle the complexities of understanding arbitrary-length videos. The model addresses the computational challenges posed by long video sequences by implementing two core techniques: Memory-Propagated Streaming Encoding and Adaptive Memory Selection. These innovations enable the model to efficiently encode long videos into a constant number of video tokens, preserving both spatial and temporal dynamics. Extensive experiments demonstrate that VideoStreaming achieves superior performance and higher efficiency on long video benchmarks, showcasing its ability to provide precise temporal comprehension for detailed question answering. The contributions include a novel approach to long video understanding, efficient encoding and memory selection strategies, and comprehensive evaluations that highlight the model’s capabilities and advantages over existing methods.
strength_weakness: strength_weakness: - Summarization Tokens: The use of summarization tokens is unclear, as it avoids attention between the TT tokens and the clip TN tokens. This design choice needs better justification and clarification.

- Small Language Model for Streaming Encoder: While the use of a small language model for the streaming encoder is interesting, there are no experiments demonstrating the necessity of language reasoning for visual-dominant dependencies (mainly visual embed. outputs). Additionally, the model is still quite large (2.7 billion parameters). A more efficient alternative, such as a shadow layer transformer, should be considered.

- Paper Writing: The proposed technical strategy is difficult to understand and lacks readability. It would benefit from clearer explanations and more detailed descriptions.

- Memory-Propagated Strategy: The memory-propagated strategy is similar to previous strategies in video understanding, such as those in [1]. The paper should differentiate its approach more clearly.

[1] Memory-Augmented Dense Predictive Coding for Video Representation Learning

- Ablation Studies: The ablation studies are not comprehensive, with only Table 8 presented. There is a lack of convincing studies on several designs, such as summarization tokens. More detailed ablation studies are needed to validate the design choices.

- Baseline Comparisons: In the main tables, the paper does not include recent instruction-tuning baselines such as LLaMA-VID. Many baselines, like LangRepo and LLoVi, are mainly modular methods based on image captioners like LLaVA, making the comparisons unfair. Including more relevant and recent baselines would provide a clearer picture of the paper's contributions.

- Progressive Training: The progressive training strategy is not innovative and can be found in previous works [1, 2]. The paper should highlight any unique aspects of its approach.
[1] LLaMA-VID
[2] A Simple Recipe for Contrastively Pre-training Video-first Encoders Beyond 16 Frames
questions: questions: - Performance on Egoschema: I am very curious about the performance on the Egoschema dataset. As it is an egocentric video dataset, and since the pretraining dataset does not include egocentric videos, how does the model achieve a significant zero-shot performance (44.1) comparable to the finetuned MC-ViT-L baseline (44.4)? Could you provide more insights or experiments that explain this result?

- Visualization in Figure 5(b): The accurate grounding along a video exceeding one hour is impressive. However, I wonder if the LLM already recognizes specific information from the query, such as "Philippe Petit" and "Twin Towers." Additionally, the grounding intervals and the text outputs do not seem to have solid associations. Could you clarify how the grounding intervals are determined and ensure that they are directly linked to the text outputs? Are there any qualitative or quantitative measures used to validate these associations?
limitations: limitations: Please refer to the weakness & questions section.
paper_summary: paper_summary: This paper introduces VideoStreaming, a Vision-Language Large Model (VLLM) designed for comprehensive video understanding. It effectively manages the challenges of computational overhead associated with processing long videos by innovatively using a constant number of video tokens. VideoStreaming demonstrates superior performance and efficiency on long video benchmarks, excelling in detailed question answering with accurate temporal comprehension.
strength_weakness: strength_weakness: 1. The core strategy still relies on processing video through key clips, which missed some relevant works, such as "LGDN: Language-guided denoising network for video-language modeling" (NeurIPS 2022).
2. The method for selecting key clips could potentially lead to redundancy, especially if multiple similar clips have high relevance scores. Integrating a regularization function to promote diversity in key clip selection could help mitigate this issue.
3. In scenarios involving multiple queries about a video, the model may need to repeatedly extract key clips for each question, potentially affecting efficiency. The paper could benefit from a discussion on optimizations that reduce the computational overhead in such use cases.
questions: questions: Please see weaknesses. I will raise my score if the author well address my concerns.
limitations: limitations: None.
paper_summary: paper_summary: This paper proposes VideoStreaming to understand arbitrary-length videos by streamingly encoding video tokens and adaptively selecting a constant number of them. Through careful designs, including a small LLM streaming encoder, prefix task, modified attention, memory propagation, and Gumbel-Topk token selection, the method successfully achieves superior performance on long video benchmarks, showcasing precise temporal comprehension for detailed question answering.
strength_weakness: strength_weakness: My main concerns are as follows: 

A. In common short/medium video length benchmarks, VideoStreaming could be unnecessary; 

B. For long video length benchmarks, the paper does not provide enough experiments; 

C. Promising experiments, like real-time interaction, are completely ignored in this paper.

Specifically,

A.1. This paper actually uses much more data during training compared to LLaVA-Next-Video:

- LLaVA-Next-Video-7B: 558K for MLP + 860K for MLP & LLM

- VideoStreaming-8.3B: 790K for MLP + 763K for small LLM + 345K for all

VideoStreaming is trained with 480K more data, which cannot be ignored. However, note that LLaVA-Next-Video-7B has slightly better results on VideoChat-GPT and even employs a much simpler pooling strategy to encode video tokens. Given that VideoChat-GPT's average duration is 1.81 minutes (see Table 1), we can infer that when the short video is less than 2 minutes, VideoStreaming may not offer advantages.

A.2. For medium-length videos, such as EgoSchema (average 3 minutes), the paper seems to have missed the results of VideoChat2 mistral-7B, which shows an accuracy of 54.4%, much higher than the 44.1% reported in the paper. Although it is not entirely fair since VideoChat2 mistral-7B uses even more training data, its model employs a simple Q-former with 16 segments and 16 frames to handle the video. Therefore, I also suspect that VideoStreaming may not present advantages in medium-length videos. Here I have omitted the discussion of MovieChat-1K since its question setting does not really relate well to minute-long videos.

B. Therefore, where VideoStreaming might truly excel is in handling very long videos. However, it is very disappointing that the paper does not present enough experiments on this. The ablations have been done on medium-length videos, and the authors do not provide any competitive baselines (with the same training data and procedure) in Table 7.

C. In fact, VideoStreaming completely ignores a promising application, which is long-term, real-time streaming QA, such as GPT-4o (concept). The method can significantly reduce the kv cache consumption during long video streaming and maintain high-speed response even if a long sequence has been input.

D. The demo shown in Figure 5(b) can be misleading. If you try the text-only Vicuna 13b (https://chat.lmsys.org/) and ask, "What challenges does Philippe Petit face in his quest to walk between the Twin Towers?" it can produce even better responses, such as "getting access to the top of the Twin Towers," "creating the perfect wire," "practicing the walk," and "keeping the walk a secret." This suggests that the question you provided may only prompt the LLM to recall its text pretraining data. I also doubt that the model can precisely select the relevant clips in a 2-hour movie, as that would be an extraordinary feat even for GPT-4o or Gemini-1.5-Pro.

Despite the above weaknesses, I still acknowledge the novelty of the methodology. I will decide on my final rating after reading the rebuttal.
questions: questions: 1. Would it be possible to use the previous layers of Vicuna to replace the extra introduced (half) Phi-2, which has 1.3B parameters that cannot be ignored?

2. Should L127-L129 be correct? StreamingLLM [a] has demonstrated the importance of initial tokens. I think it is your designed prefix task that restricts the streaming encoder from achieving this.

3. The method still needs to segment clips. Why not directly operate on the frames with a moderate FPS (e.g., 2)? This should make the method more general. Or will this also incur a huge memory cost for the streaming encoder since it is still at the billion-level?

4. The ablation in Table 8 is hard for me to understand. First, what is the performance if there is no memory and no selection? Second, the performance variance in MovieChat is so huge. Are these results from different trained models, or just the same trained model but different inference?

[a] Efficient Streaming Language Models with Attention Sinks, ICLR 2024.
limitations: limitations: The limitations are discussed in the supplementary material: (1) simply uniformly sampling frames to form a set of short clips for memory-propagated streaming encoding rather than using adaptive segmentation; (2) the number of tokens may not be sufficient to represent clips with abundant visual content and intensive temporal dynamics. These are indeed limitations and are well discussed. However, as mentioned in the 'weakness' section, most limitations pertain to the experimental setting.

Comments: title: Many thanks for the detailed response
Additional Response to Reviewer sqPz
Additional Response to Reviewer 9Ye5
Additional Response to Reviewer FUM4
Post-response
comment: Dear Reviewer FUM4:

We are delighted to hear that your concerns are addressed! Many thanks again for your very constructive comments, which have helped us improve the quality of the paper significantly. We will include the experiments and clarifications in the revised version.

Best,

Paper 6273 Authors.
I would like thank the authors for their effort for rebuttal. I have read the response from authors as well as the reviews from other reviewers. The rebuttal has resolved most of my concerns. Therefore, I decided to keep my original rating.
Dear Reviewer 9Ye5:

We are delighted to hear that your concerns are addressed! Many thanks again for your very constructive comments, which have helped us improve the quality of the paper significantly.

Best,

Paper 6273 Authors.
Dear Reviewer sqPz:

We are delighted to hear that most of your concerns are addressed! We will include the ablation study on short-term videos, the experiments on online/streaming benchmarks, and non-movie examples in the revised version.

Regarding your concerns in A5 and A6, there is no movie title at the beginning frame, only the visual tokens and question tokens are input to the LLM to produce the answer. For grounding, there exist reasonable alternatives to be selected in the movie, but the propagated memory allows the model to select a subset to answer the questions. We will present more detailed grounding results in the revised version.

Many thanks again for your very constructive comments, which have helped us improve the quality of the paper significantly.

Best,

Paper 6273 Authors.
Thank you for the detailed responses. My concerns have been addressed, and I would like to raise my score.
> A1: ...

The response somehow works for me. Let's focus more on the long-form videos. However, I do think that in a future version, you should provide an ablation study on short-term videos to show whether your method decreases performance compared to the simple pooling strategy.

> A2 & A3: ...

Nice & important results. Many thanks for that.

> A4: ...

The recommendation is to validate on some online/streaming action detection benchmarks, or follow Streaming Vid2Seq [a]/VideoLLM-online [b] on CVPR 2024. 

[a] Streaming Dense Video Captioning. arXiv: 2404.01297.

[b] VideoLLM-online: Online Video Large Language Model for Streaming Video. arXiv: 2406.11816.

> A5: ...

Now the example is less misleading. Thanks for that. However, I wonder if there are instances, such as a movie title at the beginning frame, where knowledge could still be leaked. Could you provide non-movie examples? If the rebuttal cannot provide an anonymous link, please include them in a future version.

> A6: ...

This does not convince me. I suspect the grounding is cherry-picked and there are many false positives and true negatives. However, I understand that this is what the demo needs.

> A7 & A9: ...

Thanks for that. The results are consistent with my estimate.

Overall, I appreciate the hard work of the authors. Most of my concerns have been addressed, and I plan to increase my rating score.
Dear Reviewer sqPz,

We sincerely thank you again for your great efforts in reviewing this paper, especially for the precious advice that has helped us improve the quality of this paper significantly!

In response to your suggestions, we have supplemented the experiments and discussions in the rebuttal. If there are any further clarifications or additional experiments you would like us to address, please do not hesitate to let us know.

Best,

Paper 6273 Authors.
Dear Reviewer 9Ye5,

We sincerely thank you again for your great efforts in reviewing this paper, especially for the precious advice that has helped us improve the quality of this paper significantly!

In response to your suggestions, we have supplemented the discussions and clarifications in the rebuttal. If there are any further clarifications or additional experiments you would like us to address, please do not hesitate to let us know.

Best,

Paper 6273 Authors.
Dear Reviewer FUM4,

We sincerely thank you again for your great efforts in reviewing this paper, especially for the precious advice that has helped us improve the quality of this paper significantly!

In response to your suggestions, we have supplemented the experiments and explanations in the rebuttal. If there are any further clarifications or additional experiments you would like us to address, please do not hesitate to let us know.

Best,

Paper 6273 Authors.
Dear Reviewer QjSL:

We are delighted to hear that your concerns are addressed! Many thanks again for your very constructive comments, which have helped us improve the quality of the paper significantly. We will include detailed discussions with recent works like [1] and incorporate the results on more recent benchmarks like [2].

Best,

Paper 6273 Authors.
I would like to thank the authors for their hard work and appreciate the honest acknowledgment and significant updates during the rebuttal phase. which have mainly addressed my concerns.

Additionally, I suggest the authors to include more discussion on recent works related to long video streaming optimization in their revision, such as [1], which maintaining long context with real-time efficiency. 

Moreover, recent benchmarks like [2] might offer a better evaluation choice for long videos.

[1] Chen, Joya, et al. "VideoLLM-online: Online Video Large Language Model for Streaming Video." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
[2] Fu, Chaoyou, et al. "Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis." arXiv preprint arXiv:2405.21075 (2024).

Considering these points, I would like to raise my rating to borderline accept.