VideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation
Authors: Shiwei Wu
Joya Chen
Kevin Qinghong Lin
Qimeng Wang
Yan Gao
Qianli Xu
Tong Xu
Yao Hu
Enhong Chen
Mike Zheng Shou
Venue: NeurIPS 2024 poster (2024)
Keywords: Online video understanding; effcient modeling
Primary area: machine_vision
TL;DR: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation

Abstract: A well-known dilemma in large vision-language models (e.g., GPT-4, LLaVA) is that while increasing the number of vision tokens generally enhances visual understanding, it also significantly raises memory and computational costs, especially in long-term, dense video frame streaming scenarios. Although learnable approaches like Q-Former and Perceiver Resampler have been developed to reduce the vision token burden, they overlook the context causally modeled by LLMs (i.e., key-value cache), potentially leading to missed visual cues when addressing user queries. In this paper, we introduce a novel approach to reduce vision compute by leveraging redundant vision tokens ``skipping layers'' rather than decreasing the number of vision tokens. Our method, VideoLLM-MoD, is inspired by mixture-of-depths LLMs and addresses the challenge of numerous vision tokens in long-term or streaming video. Specifically, for certain transformer layer, we learn to skip the computation for a high proportion (e.g., 80\%) of vision tokens, passing them directly to the next layer. This approach significantly enhances model efficiency, achieving approximately 42% time and 30% memory savings for the entire training. Moreover, our method reduces the computation in the context and avoid decreasing the vision tokens, thus preserving or even improving performance compared to the vanilla model. We conduct extensive experiments to demonstrate the effectiveness of VideoLLM-MoD, showing its state-of-the-art results on multiple benchmarks, including narration, forecasting, and summarization tasks in COIN, Ego4D, and Ego-Exo4D datasets. The code and checkpoints will be made available at github.com/showlab/VideoLLM-online.

LideoLLM-MoD: Efficient Video-Language Streaming with Mixture-of-Depths Vision Computation
Shiwei Wu 1:2 .Joa Chen Kevin Qinghong Lin?, Qimeng Wang Yan Gao Qianli Xu Tong Xu Yao Hu', Enhong Chen Mike Zheng Shou 2
University Science and Technology of China & State Key Laboratory of Cognitive Intelligence Show Lab, National University of Singapore *Xiaohongshu Inc. Inntiitute for Infocomm Research, A*STAR
Abstract
A well-known dilemma in large vision-language models (e.g., GPT-4, LLaVA) is that while increasing the number vision tokens generally enhances visual understanding, it also significantly raises memory and computational costs, espe- cially in long-term, dense video frame streaming scenarios. Although learnable approaches like Q-Fommer and Perceiver Resampler have been developed to reduce the vision token burden, they overlook the context causally modeled by LLMs (i.e., key-value cache), potentially leading to missed visual cues when addressing user queries. In this paper, we introduce a novel approach to reduce vision compute by leveraging redundant vision tokens "skipping layers" rather than decreasing the number of vision tokens. Our method, VIDEOLLLMMODD, mixture-of-depths LLMs and addresses the challenge of numerous vision tokens in long-term or streaming video. Specifically, for certain transformer layer, we learn skip the computation for high proportion (e.g., 80%) of vision tokens, passing them directly to the next layer. This approach significantly enhances model efficiency, achieving approximately -42% time and -30% memory savings for the entire training. Moreover, our method reduces the computation in the context and avoid decreasing the vision tokens, thus preserving or even improving performance compared to the vanilla model. We conduct extensive experiments to demonstrate the effectiveness of VIDEOLLLM-MOD, showing state-of-the-art results on multiple benchmarks, including narration, forecasting, and summarization tasks in COIN, Ego4D, and Ego-Exo4D datasets. The code and checkpoints will made available
inspired by
Introduction
Recent advanceennts in large language models [65, 66, 6, 63, 72, 73, 56, 21, particularly with GPT- 4o [62], have led many believe that the development of J.A.R. V.L.S. Al assistant is becoming increasingly feasible. Such an assistant would operate in streaming manner, remain always-on, and be multimodal to facilitate interaction with users.
While existing video-based Large Multi-modal Models (LMMs) [54, 40, 84, 88, 69, 38, 57, 89, 81, 36, 20, 47] have shown significant capabilities in general visual content understanding and reasoning, these models primarily operate in an offline setting, provide response for a few sampled frames within a video the event-level, which falls short online settings where there a need for prompt, concise, and frame-aligned answers for the continuous video frames, as shown in Figure 1. For
"Corresponding Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Reduce input vision token
2. Eorly-exit in intermediate loyer
Offiine image/video
(insooplette contents, high laterey io ottn//huster opention)
(oerforrance degrodation,, heuvy cose in seleet retuis tabrns)
Cross-atrickustee Meeging
[TEXT]
LM Head
0
die grune
Sparse frames (e.g,, sample 8 frames/clip)
Attention/cluster based merging
Prune visual tokens after k-th layer (e.g, FastVIECCV24))
(e.g, LLMMA-VID/ECCVZ2) Chat-UniVi(CVPRAA)
(keep complete sontexts, low costlltency io selecting, retain performonce compored wich full token computation)
Online video
[TEXT]
LM Head
Challenges in offline->online video
shop bynamically
#1: Keep causal context in complete
#2: Heavy cost/ lost//attecyy training/inference
#3: Performance degradation
Dense frames
Dynamically select critical vision tokens at certain layer (Ours)
(l.e., process every incoming frame)
Figure I: Comparison of efficient processing challenges in offline and online video. Online video introduces distinct challenges due to the need for real-time processing as frames stream continuously.
instance, in response to a query such as "remind me when I should add salt", the online assistant ought to evaluate each incoming frame and give temporal-aligne suggestions, taking into account historical visual and linguistic context, rather than merely summarizing the video at the event level. Consequently, shown in Figure 1, online assistants face significant computational demands and challenges as they are required to engage in causal modeling for every frame of the long video, and current approaches [9] only rely exclusively on the CLS token for each frame, limiting the vision capability to spatial understanding, which is inadequate for scenarios that require fine-grained scene understanding.
is intuitive to enhance spatial understanding by inte- grating additional pooled spatial tokens per frame. How- ever, expanding vision resolution in the online scenario is challenging. Due to the dense attention mechanism and the deep layer design of existing LLMs, the training cost, including GPU memory and training time, increases quadratically as the number of vision tokens expands (e.g.. from 0.6k~66k vision tokens for video consisting of 600 frames), which poses significant challenges to scaling up vision capabilities. Long videos, particularly online streaming videos, exhibit high redundancy in visual con- tent, which suggests that a sparser approach could be used to process visual signals, potentially reducing the need for full attention in vanilla transformer-butt based LLMs without sacrificing performance.
Training Comp-cost, ((K.) denotes #viiion tokens)
preg)
Vanilla
VideoLLM-MoD
0.60
0.55
1.5x Speedup
Tuee
0.45
GPUmem save
60 Training Memory (GB)
Figure 2: Training Computation Cost. VIDEOLLM-Mol exhibits greater effi- ciency compared to the baseline.
efficient approach scaling up vision resolution for online video large language models. Inspired by the Mixture-off of--xeertss (MoE) [21] and Mixture- of-Depth (MoD) [67] in LLMs, which utilizes conditional logic route tokens to one of many expert feed-forward networks (FFNs) or certain intermediate layers, we propose that vision tokens in specific blocks can be either routed to subsequent self-attention and FFN operations or bypassed via residual connections. This approach, which we term Wixture-of-Deptth for vision tokens, allows for the natural reduction of redundant vision information. Consequently, the model can learn which vision tokens are crucial, thereby optimizing computation accordingly. We surprisingly discovered that sparse operation at the token-level won't harm both vision capability and language modeling and is even better since it preserves original context as well as neglects the redundant vision signals, and it can dramatically reduce the training cost as shown in Figure 2, serving as "free lunch" in vision scaling. For each frame, instead of distributing FLOPs uniformly across all vision tokens in every decoder layer, we utilize a learnable module LayerExpert to allocate compute to critical vision tokens within the frame dynamically. Only a few vision tokens selected by the top-k route mechanism are processed by the following self-attention and FFN (Feed Forward Network) operations and the remains are skipped through residual connection. As shown in Figure 1, compared to directly dropping[ 68]
In this paper, we propose VIDEOLLM-MOD
2

---

vision tokens or merging [39, 20, 36] them to reduce computation, the skip within context mechanism preserves the completeness of the context, allowing for equivalent vision capability with significantly less computational effort. The proposed approach can also be generalized to traditional offline video settings seamlessly, such as COIN [71] and EgoExo4D [28] benchmarks.
We summarize our technical contributions as follows:
We propose VIDFOLLM-MODD, an efficient approach to scaling vision resolution for online VideoL.LM with reduced computational cost and similar or even better performance.
We propose LayerExpert determine which vision tokens should be processed at certain layers, leveraging the model to adaptively allocate computation to critical regions within incoming frames.
Our experiments on Ego4D, EgoExo4D, and COIN benchmarks demonstrate the effectiveness and generalizability of our VIDEOLLLMMODD
2 Related Work
Efficient Modeling in Transformer-based Models. The notorious squared complexity in vanilla Transformers [74] is a well-known problem, as it is one of the key bottlenecks scaling the sequence length. In recent large multimodal models (LMMs) [1, 39, 50, 93, 63], prefix visual tokens are used as a fixed budget for context, significantly contributing to their efficiency. This issue becomes more pronounced in online video scenarios with denser video frames. Previous studies on large language models (L7Ms)[79. 24, 10. 18] have explored the use of sparse computation to maintain performance during inference while reducing computational costs. However, these methods still incur significant training expenses. Efforts to reduce training costs through token pruning[68] and merging [36] techniques are not suitable for online scenarios, they require offline computation directly reduce token levels. Mixture-of-Dee -epth [67] investigates the allocation of computation across model depth for language tokens, balancing performance with speed. Our model, this approach to online video. We found that reducing vision computation in the context across model depth not only maintains but can even improve performance by removing high redundancy in video.
extends
Large Multimodal Models for Online Video Understanding, Inspired by the success of numerous large language models (LLMs) [6, 63, 61, 85, 86], a series of large multimodal models (LMMs) [I. 39, 50, 93, 15] have subsequently been developed to further enhance our comprehension of the world. capable of addressing variety of standard benchmarks in video understanding, including temporal action localization [51], and video dialogue and question answering [40, 69, 54, 88, 44], while also demonstrating strong potential in broader multimodal applications [12, 13, 11, 46, 76, 77, 78]. However, while these models analyze entire video frames to make predictions in "offline" setting, they are not optimized for real-time applications such as augmented reality (AR) glasses and autonomous driving systems. In light of this growing demand, benchmarks for online scenario such as action detection [90, 75] and anticipation [26, 89], which are designed to interpret events at the current timestamp without access to future data, are becoming increasingly critical. VideoLLMM-nnlne [9] serves as the first attempt to build an assistant using LLMS in an online video scenario. However, its spatial capabilities are limited, as it uses only single CLS token to represent each frame, and expanding the vision scale is computationally expensive. VIDEOLLM- proposes an efficient approach to scaling vision resolution by reducing vision computation in context, thereby enhancing spatial ability without incurring high computational costs.
Current large multimodal models (LMMs)
Scaling up Vision Resolution for Large Multi-modal Models. Scaling up the visual resolution for LMMs is effective approach enhancing vision capabilities. By utilizing 5x more vision tokens compared to LLaVA-1.5 [19], LLaVA-NeXT [49] achieved improved vision understanding. However, scaling vision tokens in online video scenarios presents significant challenges, as the training cost increases quadratically with the expansion of vision tokens, requiring the processing of every incoming frame in long videos. To handle long-context vision tokens in LMMs. CogAgent [31] integrates high-resolution image features into a low-resolution pathway via cross-attention across decoder layers. LLaMA-VID [20] utilizes antexx-attentionn represent each frame with two key tokens. Both approaches are only applicable for offline video, as the high latency induced by the additional cross-attention mechanism is unacceptable in online scenarios. In contrast, VIDEOL.LM-M receives streaming video-language input continuously and can reduce computation efficiently during every forward pass without additional overhead. This enables temporal- aliinned responses, making suitable for real-time applications.
3

---

3
Method
In this section, we introduce our VIDEOLLM- MOD framework, an efficient approach training an online video large language model with a larger vision resolution.
3.1 Model architecture.
We depict the overall model architecture shown in Figure drawing parallels to LLaVA [50, 19, 49] in its design. The model is composed of three principal components: an image encoder, an MLP projector, and language model. Each video frame embedding is represented (1 + hp wip) c, which denotes the CLS token and the average pooled spatial tokens. The frame embeddings extracted by the image encoder are subsequently processed through the MLP projector frame tokens. These tokens are interwoven with language tokens, forming the input for an LLM. We incorporate LoRA [32] in every linear layer of the LLM for efficient tuning. To select the most critical vision tokens, certain layers are also equipped with LayerExpert module, as detailed in Section 3.2.
Following VideoLLM-onnln [9]. in addition to the language modeling (LM) loss, also utilize an additional streaming loss to ensure the model remains silent when is unnecessary to output responses. Both training objectives employ cross-entropy loss follows:
N N
L =
lit1 log Pj p Jg log P [BDS]
LMLoss
StreamingL.oss
where 15 and 8j are condition indicators: 15 is if the j-th token is a language response token, and O otherwise; 8j is if both (1) the j-th token the last token of a frame?, and (2) 1リ-1 = 0. The streaming EOS loss is applied to frames prior responding. P, ( represents the probability associated with the (j + 1) text token, as output by the language model head for the j-th token, while p.eeos] indicates that probability for the EOS token. The two objectives are balanced using the streaming loss weight a.
3.2 Scale up Vision Resolution for Online Video.
The performance of online assistants improves with increased vision scale (i.e., (1 + hp x Wp)). However, enhancing vision resolution in online scenarios is challenging because the number of vision tokens grows with video duration, leading to quadratic computational complexity. As shown in Table 7, online videoLLMs must process every frame of long videos during both training and inference maintain the integrity of the complete visual and linguistic historical contexts, which places significant demands on GPU memory and computational resources.
Motivation.
We hypothesize that videos exhibit high redundancy in temporal and spatial since consecutive frames often share a large portion of their content, especially if the frames are captured in quick succession. Just as humans continuously "see" their surroundings without always "focusing" on every visual detail, it is intuitive that we should skip some vision tokens in certain layers when processing them with deep model.
Selecting vision tokens via LayerExpert in certain block. We leverage per-block LayerExpert module to learn the selecting/routing behavior, i.e., learn which vision tokens require more or less processing than others. The LayerExpert identify the "importance score" (in scalar weights) of each vision token within a frame, and only the top-k vision tokens are processed by the following operations. Notably, since the vision tokens of different frames in the online streaming scenario are processed in a causal manner, the top-k selection is performed at the frame level, meaning the top-k vision tokens are selected within each frame. The language tokens are always processed since they are much less redundant and significantly fewer in number compared to vision tokens.
Specifically, suppose we have a sequence of length N interleaved with n, language tokens and ny vision tokens. For the given layer I, the sequence X = [Interleaved(2]] [2), xx. < t, S n, 1 < 8 S }. Within the (1 + hp x wp)) vision tokens of each frame, the LayerExpert determines the
p Loss is applied only to the last token when frame consists of multiple patch tokens.

---

(1) Full computation
(2) Early exit
(3) Top-k route via LayerExpert
LM Head
LM Head
LM Head
Layer i+1
Layer
Layer i-1
vision tokens per frame
vision tokens per frame
vision tokens per frame
Computed vision token
Vision Embed
Text token
Pooled to NxN shape
Skipped token
timestamp
Figure 3: VIDEOLLM-MOL selects the top-k vision tokens within each frame in certain layers via LayerExpert. We observe that performance drops dramatically with Early-exil as critical vision tokens miss subsequent processing. By retaining crucial vision tokens in certain layers and reducing redundant tokens that may mislead understanding, VIDEOLLM-MOD achieves better performance with significantly lower computation costs compared to coll-computation baseline.
importance score μ for a given vision token using a linear projection μ4: = wh d... Then, vision tokens are selected based on vision keep ratio r for following processing, and P/ is the (1 - r)-th percentile among the weights f of frame vision tokens. The block's output for the given vision token as follows:
Mez fe: b
if Uv: > pd if yf S P
1+1
where the X represents the interleaved tokens consisting of all language tokens and top-k vision tokens in layer I, and denotes the subsequent self attentttnnion and the FFN operations.
3.3 Efficiency analysis of VIDEOLLM-MOD.
We further analyze the computation cost of our approach. Except for the decoder layers, the other modules, including LayerExpert, the MLP projector, and LoRA, are fixed given certain inputs and are significantly smaller than the decoder layers of the language model. Therefore, we ignore their FLOPs computation and only consider the computation of the multi- nuat attention (MHA) and feed-forward network (FFN) modules in the FLOPs estimation.
Suppose the language model has L total hidden layers, in which and denote the hidden size dimension, and the intermediate size of FFN, respectively. The input sequence is interleaved with n, vision tokens and n, language tokens. We insert LayerExpert in K layers with vision keep ratio inside of the entire decoder layers. For each attention head, the theoretical FLOPs of the layer with LayerExpert is:
FLyePenyertspeen 4(m rn.) + 2(n; + rn.)) 2(n, + rn,)dm,
while = in vanilla transformer layers. Since in the online video setting, vision tokens are significantly more than language tokens, i.e., n, > n4, the FLOPS of entire decoder proportional to the vision keep ratio and the number of layers equipped with LayerExpert follows:
5

---

200 400 600 860 1000 1200 1400 Allocated GPU Memory during Inference
(00M)
Computation-cost vision keep ratio
45
1.7x Longer
context
Vanilla
GPU Memory (GB)
40
45
35
Training FLOPs
40
0.6x Fl.PPs
35
25
Vanilla VideoLLM-MoD
30
VideokLM-MoD (Default)
20
15
0.0.1102003344055000000000000000
Duration of Real-Time Video (seconds)
(a) Calculated training FLOPs vision keep ratio r. By default, VIDEOLLM-MOD only requires 0.6x FLOPs compared to the Full-
(b) Allocated GPUMemory in inference phase.
By saving the KV cache of historical states, VIDEOLLM-MOD supports 1.7x longer vidco than baseline.
computation baseline.
Figure Efficiency analysis of VIDEOL.LM-MOD in both training and inference phase.
FLOP Sdccoder
FLOPs Si LaycrEppnnp
+
FLOPSyanill
(4)
A=
k=
We further calculate the total FLOPs* of VIDEOLLM-MOD and the Full-computation baseline during the training phase in real-world scenario. As shown in Figure 4a, the practical FLOPs of VIDEOLLM-MOD is only 0.6x that of the baseline, and this value can be further reduced the vision scale of each frame is larger, demonstrating the excellent scalability of our approach.
By skipping redundant vision tokens certain layers, VIDEOLLM MMD not only reduces training computation costs but also improves inference efficiency. As shown in Figure 4b, reducing the intermediate VCcache in historical states allows us to support 1.7x longer context and achieve a comparative inference speed compared to baseline, facilitating deployment in real-world applications.
Experiments
4.1 Experimental Settings
Datasets. We validate the effectiveness of proposed VIDEOLLM-MoD on both online and offline settings, including egocentric video dataset Ego4D [27] and EgoExo4D [28], as well as instructional video dataset COIN [71].
Ego4D Narration Stream Benchmark: Following VideoL.LMoonline [9], we utilize the dense Ego4D timestamp-narration to create a streaming set, aiming to generate timely narrations similar to those produced by Ego4D human annotators [27].
Ego4D long-term action anticipation (LTA) Benchmark:
This benchmark requires predicting the next Z = 20 actions (verbs and nouns) for a given vidco based on the previous steps. Following previous studies [89, 33], use the standard Ego4D v2 splits,
This task involves recognizing fine-grained key steps from procedural egocentric videos during the test phase, using models that can leverage multiple time-synchronized views during training.
ExoExo4D Fine-grained Keystep Recognition Benchmark:
COIN Benchmarks: Following previous works [48, 91, 58, 83]. we evaluate our model on six common benchmarks of the COIN dataset: step recognition, step forecasting, task summarization. procedure forecasting, and procedure forecasting with goal.
Calcullated via DeepSpeed FLOPs Profiler, processing 600 frames with (1+3x3) patches on single NVIDIA A100 GPU. The layers with LayerExpert are interleaved in every other layer in default.

---

Evaluation metries and implementation details: For online benchmark, following VideoLLM- online [9], we use the Language Modeling Perplexity (LM-PPH) and AM-Correctmess to evaluate the language modeling capability at the given timestamp. To evaluate the temporal alignment capability as an online assistant, we use the Time Difference (TimeDifff) and Fluency to comprehensively evaluate both the language modeling and temporal effectiveness. We trained all models on 8x NVIDIA A100 GPUs. For each module, we use SigLIP-L//6 [87] as the visual encoder, a 2-layer MLP as the multimodal projector, and Meta-Llama-3-83
[56] for the language model. For the vision embedding of each video frame, we use (1 + 3 x 3) tokens (CLS token + averaged pooled spatial tokens), with frame rate of FPS. We add LoRA [32] to all linear layers of the language model with rank of 128 and a scaling factor of 256. Additional details can be found in Appendix A.3. For the rofe between computation cost and performance, we insert LayerExpert every other layer and set the keep ratio r to 0.2 as the default setting.
4.2 Online Experiments
Frame Strategy
Training Cost Speedup
Ego4D Narration Stream Validation Set
Method
FLOPs
IM-PPI.4
TimeDiff Fluencyt IMM-orrectnesst
LideoLLM-oolliee [9]
5.75T
8hrs & n/a
2.43
2.04
45.1%
48.1%
Full-computation
48.29T
24hrs & n/a
2.40
2.05
45.3%
49.0%
FarlyFxit
7.14T
10hrs & 2.4x
2.50
2.29
41.3%
46.2%
1+3x3
LayerSkip
26.35T
13hrs & 1.8x
2.52
2.24
42.0%
46.5%
VIDEOLLM-MOD
30.74T
14hrs & 1.7x
2.41
2.04
45.2%
48.9%
Table 1: Online experiments on the Ego4D narration benchmark.
VIDEOLLM-MOD achieves
comparable metrics to the Full-computation baseline with less computation cost.
We compare our VIDEOLLM-MOt model with various baselines on the Ego4D narration benchmark, as shown in Table We analyze the baselines in detail as follows.
VideoLLLM-online [9]. For a fair comparison, we re-implemented the using the same visual encoder and language model as VIDEOLLM-MoN in all experiments. By embedding each frame using only the CLS token, VideoLLM-onlin [9] achieves slightly worse performance on this benchmark due to the relatively simple narration, which does not heavily rely on fine- the-igr vision. Moreover, we found that larger vision resolution can indeed benefit performance, as shown in Figure 5, and in experiments that demand more detailed visual information as shown in Table 4, 5.
[9] baseline
Using vanilla transformer architecture, all vision tokens are processed densely across every layer, which significantly increases the training cost.
Full-computation.
ExrlyExit. Building on studies of language-only LLMs [25, 17]. we adapt this approach to the online video setting. All vision tokens are processed in the shallow layers, then skipped in the deeper layers (equivalent to VIDEOLLM- MoD with layers). Empirically, we found that early exit at Layer 2 offers the best tradeoff between performance and computational cost, also highlighted in previous studies [79,, 10]. This approach shows the lowest computation but the worst performance, as it misses most of the vision information.
1 in the first few layers and r 0 in the remaining
LayerSkip. Introduced in previous LLM studies [18], we adapted the approach to the online scenario, skipping all vision tokens in every other layer (treated as VIDEOLLM-MOO interleaving layers with r = 1 and 0). Compared with VIDEOLLLM-MDD. the performance drops significantly as critical vision tokens miss processing in certain layers.
Our VIDEOLLM-MON exhibits the best tradeoff' in online video scenarios, significantly reducing computational costs when processing excessive frames without sacrificing performance compared to the Full- colputaomtttioon baseline. Moreover, we discovered that our approach performs better than the Full- coll-computation baseline in practical use, as shown in Figure 5. seems counterintuitive that fine-tuning LLM with MoD performs better than using the vanilla model. The dynamic layer skipping methodology of the former results in less vision computation during the forward process, which is likely to weaken the spatial understanding capability. However, we argue that this increases the learning difficulty, as it forces the MoD gate at each layer to focus on the important vision tokens in current causal contexts. This may reduce the risk of overfitting and learn more robust model.
7

---

4.3 Ablation Study
Insertion strategy of LayerExpert.
Table 2 presents the ablation study results for the insertion strategies of our LayerExpert. We constructed different settings for inserting LayerExpert in the transformer layers. All and Interleaved refers insert LayerExpert in ev- erylevery other layer. The Interleaved strategy demonstrates a better trade-off between computation cost and performance.
Insertion strategy Fluencyt
FLOPs
All 42.5%
47.5%
13.18T
All-Deep 44.8%
48.8%
15.37T
Interleaved
45.2%
48.9%
30.74T
Interleaved-Deep
45.2%
49.0%
31.83T
The postfix -Deep denotes that vision token skipping is performed only in the deep layers (i.e.. layers af- ter Layer 2). Previous studies (79, 10] indicate that attention allocation across all tokens in the shallow layers (the first two layers) is much more balanced compared to the deep layers, making these shallow layers more vulnerable to token skipping. Our results with and without -Deep also indicate this phenomenon.
Table 2: Ablations the insertion strategy of LayerExpert in transformer layers. The Inter- leaved strategy strikes the best trade-off among the variants.
Selecting the critical vision token. As shown in Table 3, to validate the necessity and effectiveness of allocating computation to the crucial vision tokens, we created two variants that select vision tokens either randomly uniformly. The poorer performance on TifeDiff indicates that the online capability significantly impacted by missing critical vision information. This suggests that determining which vision tokens deserve processing is essential for maintaining performance while reducing redundancy.
We also conducted ablations on the number of vision tokens retain based on the vision keep ratio Even with rela- tively fewer tokens and FLOPs, VIDBOLLM-MON achieves sat- isfactory results, further demon- strating the critical vision selec- tion capability of LayerExpert and highlighting the high redun- dancy present in the video.
Keep Strategy
Keep Ratio
Ego4D Narration Stream Validation TimeDiff's
LM-PPL,
Fluencyt
LM-Correctenstt
FLOPs
Random
y= 0.2
2.18
43.6%
48.1%
30.74T
Uniform
0.2
2.42
2.17
43.9%
48.6%
30.74T
r 0.1
2.43
2.11
44.7%
48.
28.54T
Learable
r 0.3
2.41
2.05
44.9%
48.7%
32.93T
Table 3: Ablations on different vision selection strategies. Choos- ing which vision tokens process crucial for efficient vision computation allocation.
4.4 Offline Experiments
We demonstrate the gener- alizability of our proposed ViDDOLLM-MoD on tradi- tional offline video scenarios, including recognition, sum- marization, and forecasting tasks. As shown in Table 5a, our method achieves the best performance compared end-to-end baselines on the Ego4D LTA benchmark, with results only slightly lower than Palm [33] and AntGPT [89], which utilize EgoVLP [45]
Method
use HowTo100M
Renchmark Top-1 Accuracy?
Step
Task
Next
Proc.
Proc.+
ClipBERT [37]
V
30.8
65.4
TimeSformer
17.0
Paprika [92]
X
51.0
85.8
43.2
DistantSup [48]
X
54.1
90.0
39.4
41.3
VideoTF [58]
x
56.5
91.0
42.4
40.2
46.4
PracedureVR1. [91]
x
56.9
90.8
46.8
VideoTaskGruph
57.2
90.5
40.2
AidooLL-oonlne [9]
62.5
92.2
49.3
48.6
53.3
(Pull-computation))
Ours
63.4
92.8
49.7
49,8
53.3
Table 4: Results on COIN benchmarks (left to right): step recog- nition, task recognition, next forecasting, procedure forecasting, procedure forecasting with goal.
pretrained features followed by cascading performance enhancement methods.
By expanding the vision resolution, VIDEOLLM- achieves state-of-the-arr performance, sig- nificantly surpassing VideoLLM-on [9], which only uses the CLS token for each frame. This is particularly evident in tasks requiring complex spatial context understanding, such as the EgoExo4D Fine- gra-ngr Keystep recognition benchmark [28], as shown in Table 5b. Furthermore, our method achieves the best performance on most of the COIN benchmarks [71], as illustrated in Table even outperforming our coll-computaiion baseline. By adaptively focusing processing critical vision

---

Ego Accuracy (%)
Method
Eg
Not use
End-to
-end?
Ego4D LTA ED Z=20,
Verb Noun
Action
Method
Train data (v1)
on Val (v1/v2)
TimeSFormer |5| (K600)
ego
35.25/35.13
CLIP [16]
0.739 0.769
0.941
TimeSFormer [5] (K600)
ego,
32.67/32.66 68
Ego [82]
0.722 0.764
0.935
EgoVLPv2 [64] (Ego4D)
ego
36.89/36.51
I-CVAE [55]
0.753 0.749
0,931
EgoVLPv2 [64] (Ego4D)
ego,
37.03/35.88 84
Ours
Vi.LLM-o. [9]
HierVL [3]
VideoLLM M [8]
0.724 0.735
0.689 0.676
0.691 0.692
0.721 0.725
0.928
0.889
0,921
0.884
EgoVLPv2 [64] (EgoExo)
EgoVLPv2 [64] (EgoExo)
VI Encoder [59] (EgoExo)
ego.
ego,
ego
37.61/36.04
38.21/39.
40.23/40.34
Viewpoint Distillation [30]
ego,
37.79/38.19
Palm [33]
0.696 0,651
0.886
Ego-Exo Transfer MAE [42]
ego, eXO
36.71/37.17
AntGPT (89)
0.650 0.650
0.877
VideoLLM-one [9]
ego
40.53/40.73
Ours
ego
44.85/42.62
(a) Results on Ego4D LTA benchmark, evaluated on public server. ED@ denotes editing distance for Results EgoExo4D Fine-grained Keystep Recog- future actions.
nition benchmark.
Table 5: Experiments on COIN [71] and Ego4D [27] benchmarks. VIDEOLLM-MOD achieves best results among end-to-end models.
tokens, VIDEOL.LMMMDD not only reduces computation but also excels in spatial-temporal scene understanding, particularly in complex contexts.
4.5 Experiments on General Benchmarks
We further validate the effectiveness of our approach on general benchmarks shown in Table 6,7, and 8. Our VIDEOLLM-MOD was trained using the same default settings, excluding streaming loss, and the same pretraining and fine-tuning data as LLaMA-VIL LIDA [20].
Training Cost Speedup
Performance (% Overall Short Medium Long
Model
Frame strategy
VidcoLLaMA [88]
frames 32 tokens/frame
65hrs 2.95x
47.9
56.0
45.4
42.1
Chat-UniVi-vil [36]
64 frames & 112 tokens/frame
53hrs 2.41
40.6
45.7
40.3
35.8
Vidco-LLaVA [44]
frames & 49 tokens/frame
60hrs & 2.73x
39.9
45.3
38.0
36.2
Ours
1fps & 10 tokens/frame
22hrs & n/a
49.2
58.4
46.6
42.4
Table Results on Video-MME [23] benchmark. Despite requiring significantly less training time,
VIDEOLLM-MOD still achieves top-tier performance.
Training Cost
Image benchmarks
Method
TFLOPs
(Pretrain + Finetune)
GQA [34]
MME [22]
POPE [43]
SQA [53]
LLaMA- VID [20]
9.8
4hrs 48.5hrs
64.3
1521.4 .4
86.0
68.3
Ours
5.8
2hrs 8.5hrs
62.8
1505.5
85.5
70.2
Table 7: Results on general image benchmarks.
Training Cost
Video benchmarks MSRVTT-QA [80]
Method
(Pretrain + Finetune) & Speedup
MSVD-QA
ActivityNet-QA [29]
Acc
Score
Acc
Score
Acc
Score
LLaMA- VID [20]
30hrs
69.7
57.7
47.4
3.3
Ours
8hrs + 14hrs & 0.56x
78.5
3.9
65.3
3.6
53.4
3.4
Table 8: Results on general video benchmarks.
4.6 Visualization of VIDROLL.M-MOD.
We observe that VIDEOLLM-MOI performs more robustly compared to the VideoLLM-onnle [9] and Full-computation baselines. By selecting critical vision tokens, the learning difficulty increases, as this approach forces the MoD gate each layer to focus important vision tokens within current

---

causal contexts. This strategy may reduce the risk of overfitting, thereby resulting in more robust model shown in Figure cases.
Streaming Video Input
Time
Time = 1.
Time
Time:
Time
Time 12.0s
Time
VideotLM-online [CUPR24)
ahat doing now?
currently slicing the bell peoper.
slicing bell peppor. Now, you're adding to the pan....
them with neodies??
You're already right you've added the bell popper to pan, you can start cooking it with noedies...
Only 215 token per frame, not enough capability
[5ient)
X
VideoLLM-onllin+ More tokens per frame, improved spobal but computational
What am doing now?
You're currently cutting the broccal
How cook them with
You're re already cooking the pasta, Just add io the vegetables and stir well You add
(missing response)
[Sdent]
VideoLLM-MoO (Ours)
You're currently outting broccal stem.
You finished cutting to move to step.
How about simple broccoli noodle stirfry?
Similar
doing now?
nondles??
according the package instructions..
[slent]
but much less
computational cost
(a) Using only CLS token often results in spatial understanding emors, e.g. mistaking 'broccoli' for 'bell pepper.' VIDEoLLM-MON improves fine-grained spatial ability by integrating more spatial tokens while reducing computation costs compared to the improved baseline. Text in red indicates incorrect response.
Streaming Video Input
Time
Videotit-oniine
Unnrooved Baseline)
More takens per
[Misdin Response)
13
(ilent)
frmm tn improse
spatial rapabilly but
lecs afficent
VidcolLM McD (Ours)
adjust
You walk around.
You pick the
the
More tokens per
frame to Improve
(Seen)
spanial capability.
Improve officiency
(b) We found that VIDEOLLM-MoU effectively reduces hallucinations and performs more robustly than the model trained with full computation. For instance, model correctly recognizes "pick up the box" while the baseline mistakenly identifies as "pick up tire."
Figure 5: Cases of VIDEOLLM-MeR on Ego4D GoalStep [70] video data.
5 Conclusion, Limitations, and Broader Impacts
In this paper, we introduced VIDEOL.LM-MOD, which scales vision resolution for video large language models in online video through efficient select critical video tokens LayerExpert. Our model can significantly reduce computational costs and memory usage with similar or even better performance compared with Full-computation baseline. Experiments on Ego4D, EgoExo4D, and COIN benchmarks confirm its efficacy and generalizability, making VIDEOLLM-MOD robust solution for online video applications.
Limitations, As our primary focus was on developing an online assistant for ego-centric or instruc- tional scenarios, we did not conduct extensive explorations exo-centric video datasets.
Broader Impacts. Beyond the online scenario, we hope our work can provide insights and contribute to general video understanding tasks, particularly those involving long videos.
10

---

Acknowledgments and Disclosure of Funding
We thank Zhaoyang Lv from Reality Labs Research, Meta, as well as Ziteng Gao, Zechen Bai, and Zongbo Han from Show Lab, NUS, for their invaluable discussions training VIDEOLLM-MoL This work was supported part by the grants from National Science and Technology Major Project (No. 2023ZD0121 104), National Natural Science Foundation of China (No. 62222213, 62072423), and Program China Scholarship Council (No. 202306340095), Mike Shou does not receive any funding for this work.
References
[1] Jean-Baptiste Alayrac. Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring. Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooci, Marianne Monteiro, Jacob L.. Menick, Sebastian Borgeaud, Andy Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karén Simonyan. Flamingo: a visual language model for few-shot learning. In NewrlPS, 2022.
[2] Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean Captitap Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap. Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Is- ard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïl White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A family of highly capable multimodal models. arXiv:2312.1 1805, 2023.
[3] Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, and Kristen Grauman. Hiervl: Learning hicrarchical video-language embeddings. In CVPR, pages 73066-23078, 2023.
[4] Kumar Ashutosh, Santhosh Kumar Ramakrishnan, Triantafyllos Afouras, and Kristen Grauman. Video-mined task graphs for keystep recognition in instructional videos. In NeurIPS, 2023.
[5] Gedas Bertasius, Heng Wang. and Lorenzo Torresani. Is space-time attention all you need for video understanding? In ICML, 2021.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Merbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, llya Sutskever, and Dario Amodei. Language models are few-shot learners. In NeurIPS, volume 33. pages 1977, 2020.
[7] David Chen and William B Dolan. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pages 190-200, 2011.
[8] Guo Chen, Yin-Dong Zheng, Jiahao Wang, Jilan Xu, Yifei Huang, Junting Pan, Yi Wang, Yali Wang, Yu Qiao, Tong Lu, et al. Videollm: Modeling video sequence with large language models. arXiv preprint arXiv:2305. 1392,, 2023.
[9] Joya Chen, Zhaoyang Lv, Shiwei Wu, Kevin Qinghong Lin, Chenan Song, Difei Gao. Jia- Wei Liu, Ziteng Gao, Dongxing Mao, and Mike Zheng Shou. Videollm-online Online video large language model for streaming video. In CVPR, 2024.
[10] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision- laniun-language models. arXiv preprint 24X:v:2403.0766 2024.
11

---

[11] Liyi Chen, Zhi Li, Weidong He, Gong Cheng, Tong Xu, Nicholas Jing Yuan, and Enhong Chen. Entity summarization via exploiting description complementarity and salience. IEEE Transactions on Neural Networks and Learning Systems, 34(1 (4(11))8297-8830 2023.
[12] Liyi Chen, Zhi Li, Tong Xu, Han Wu, Zhefeng Wang, Nicholas Jing Yuan, and Enhong Chen. Multi-modal siamese network for entity alignment. In Proceedings of the 28th ACM SIGKDD
conference on knowledge discovery and data mining, pages 118-126, 2022.
[13] Liyi Chen, Ying Sun, Shengzhe Zhang, Yuyang Ye, Wei Wu, and Hui Xiong. Tackling uncertain correspondences for mudti-modal entity alignment. In Proceedings of the 38th Conference on Neural Information Processing Systems, 2024.
[14] Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Longlora: Efficient fine-tuning of long-context large language models. arXiv preprint arXiv:2309. 12307, 2023.
[15] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C.H.Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv:2305.0655 2023.
[16] Srijan Das and Michael S. Ryoo. Video + clip baseline for ego4d long-term action anticipation. arXiv:2207.000 200,079 2022.
[17] Luciano Del Corro, Allie Del Giorno, Sahaj Agarwal, Bin Yu, Ahmed Awadallah, and Sub- habrata Mukherjee. Skipdecode: Autoregressive skip decoding with batching and caching for efficient Ilm inference. arXiv:2307.02622 2023.
[18] Mostafa Elhoushi, Akshat Shrivastava, Diana Liskovich, Basil Hosmer, Bram Wasti, Liangzhen Lai, Anas Mahmoud, Bilge Acun, Saurabh Agarwal, Ahmed Roman, et al. Layer skip: Enabling early exit inference and self- self-speculative decoding. arXiv preprint arXiv:2404.16710, 2024.
[19] Haotian Liu et al. Improved baselines with visual instruction tuning. In CVPR, 2024.
[20] Yanwei Li et al. Llama-vid: An image is worth 2 tokens in llss. In ECCV, 2024.
[21] William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, pages 1-39. 2022.
[22] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, and Rongrong Ji. MME: A comprehensive evaluation benchmark for multimodal large language models. 2023.
[23] Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang. Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint arXiv:2405.21005 2024.
[24] Suyu Ge, Yunan Zhang, Liyuan Liu, Minjia Zhang, Jiawei Han, and Jianfeng Gao. Model tells you what to discard: Adaptive ky cache compression for llms. arXiv:2310.01800, 2023.
(25) Mor Geva, Avi Caciularu, Kevin Ro Wang, and Yoay Goldberg. Transformer feed-forward layers build predictions by promoting concepts in the vocabulary space. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang, editors, EMNLP, pages 30-45, 2022.
[26]
Rohit Girdhar and Kristen Grauman. Anticipative video transformer. Arxiv, 2106.02036, 2021.
[27] Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Ro- hit Girdhar, Jackson Hamburger, Hao Jiang. Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Sean Crane, Tien Do Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina González, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jáchym Kolár, Satwik Kottur,
12

---

Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro. Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang. Xindi Wu, Takuma Yagi. Ziwei Zhao, Yunyi Zhu, Pablo Arbeláez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishma Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard A. Newcombe, Aude Oliva, Hyun Soo Park, James M. Rehg. Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3, 000 hours of egocentric video. In CVPR, pages 18973-18990, 2022.
[28] Kristen Grauman, Andrew Westbury, Lorenzo Torresani, Kris Kitani, Jitendra Malik, Triantafyl- los Afouras, Kumar Ashutosh, Vijay Baiyya, Siddhant Bansal, Bikram Boote, Eugene Byrne, Zachary Chavis, Joya Chen, Feng Cheng, Fu-Jen Chu, Sean Crane, Avijit Dasgupta, Jing Dong, María Escobar, Cristhian Forigua, Abrham Gebreselasie, Sanjay Haresh, Jing Huang, Md Mo- haiminul Islam, Suyog Dutt Jain, Rawal Khirodkar, Devansh Kukreja, Kevin J. Liang, Jia-Wei Liu, Sagnik Majumder, Yongsen Mao, Miguel Martin, Effrosyni Mavroudi, Tushar Nagarajan, Francesco Ragusa, Santhosh Kumar Ramakrishnan, Luigi Seminara, Arjun Somayazulu, Yale Song. Shan Su, Zihui Xue, Edward Zhang, Jinxu Zhang, Angela Castillo, Changan Chen, Xinzhu Fu, Ryosuke Furuta, Cristina González, Prince Gupta. Jiabo Hu, Yifci Huang, Yiming Huang, Weslie Khoo, and et al. Ego-exo4d: Understanding skilled human activity from first- and third-person perspectives. arXiv:2311.182555 2023.
[29] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: large-scale video benchmark for human activity understanding. In CVPR, pages 961-970, 2015.
[30] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv: 1503.02311, 2015.
[31] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. arXiv preprini arXiv:2312.088: /2.08914. 2023.
[32] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0966 2021.
[33] Daoji Huang, Otmar Hilliges, Luc Van Gool, and Xi Wang. Palm: Predicting actions through language models @ ego4d long-term action anticipation challenge 2023. arXiv: 16545, 2023.
[34] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6709, 2019.
[35] HuggingFacee4, Introducing idefics2: A powerful 8b vision-language model for the community. httpee/cco//blole/ee/eeeeeeoeeee ics2, 2024.
[36] Peng Jin, Ryuichi Takanobu, Caiwan Zhang, Xiaochun Cao, and Li Yuan. Chat-univi: Unified visual representation empowers large language models with image and video understanding. arXiv preprint arXiv:2311.0804 2023.
[37] Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L. Berg, Mohit Bansal, and Jingjing Liu. Less is more: Clipbert for aadlo-and-languaaa learning via sparse sampling. In CVPR, pages 7331-7341, June 2021.
[38] Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: multi-modal model with in-context instruction tuning. 230.0:2000 2023.
[39] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint 200. 2023.
13

---

[40] KunChang Li. Yinan He, Yi Wang, Yizhuo Li. Wenhai Wang, Ping Luo, Yali Wang. Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding, arXiv:2305.065 06355, 2023.
[41] Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. arXiv preprint arXiv:2311.
2023.
[42] Yanghao Li, Tushar Nagarajan, Bo Xiong. and Kristen Grauman. Ego-exo: Transferring visual representations from third-person to first-person videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6943-6953, 2021.
[43] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305. 10355, 2023.
[44] Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv:2311.0012) 2023.
[45] Kevin Qinghong Lin, Alex Jinpeng Wang, Mattia Soldan, Michael Wray, Rui Yan, Eric Zhong- cong Xu, Difei Gao, Rongcheng Tu, Wenzhe Zhao, Weijie Kong, Chengfei Cai, Hongfa Wang. Dima Damen, Bernard Ghanem, Wei Liu, and Mike Zheng Shou. Egocentric video-language pretraining. arXiv:2206001670, 2022.
[46] Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shraman Pramanick, Difei Gao, Alex Jin- peng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video-language temporal grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2794-2804 2804, 2023.
[47] Kevin Qinghong Lin, Pengchuan Zhang, Difei Gao, Xide Xia, Joya Chen, Ziteng Gao, Jinheng Xie, Xuhong Xiao, and Mike Zheng Shou. Learning video context as interleaved multimodal sequences. arXiv preprint arXiv:2407.21757, 2024.
[48] Xudong Lin, Fabio Petroni, Gedas Bertasius, Marcus Rohrbach, Shih-Fu Chang, and Lorenzo Torresani. Learning to recognize procedural activities with distant supervision. CVPR, pages 13843-13853, 2022.
[49] Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024.
[50] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurIPS, 2023.
[51] Ruyang Liu, Chen Li, Haoran Tang, Yixiao Ge, Ying Shan, and Ge Li. St-llm: Large language models are effective temporal learners. arXiv preprint arXiv:2404.00 003000 2024.
[52] Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint arXiv:2403.055: 2024.
[53] Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35:2507-2521, 2022.
[54] Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video-chatgpt: Towards detailed video understanding via large vision and language models. wwXiv:2306.05424 2023.
[55] Esteve Valls Mascaro, Hyemin Ahn, and Dongheui Lee. Intention-conditionn long-term human egocentric action anticipation. In WACV, pages 6037-6046, 2023.
[56] Meta. Build the future of ai with meta Ilaaa 3. https: wtt.llaaammmmm....mmmm hom.l....mm..... wtt../oooo////.//./a 2024.
[57] Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang. Mingyu Ding, Jun Jin, Bin Wang. Jifeng Dai, Yu Qiao, and Ping Luo. Embodiedgpt: Vision-language pre-training via embodied chain of thought. 2305. 15021, 2023.
14

---

[58] Medhini Narasimhan, Licheng Yu, Sean Bell, Ning Zhang, and Trevor Darrell. Learning and verification of task structure instructional videos. arXiv:2303.135773 2023.
[59] Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint
2018.
[60] OpenAl. GPT-4 technical report. arXiiv23003.08774,20000 2023.
[61] OpenAl. Introducing chatgpt. http:s//ppenaa..cc///gggggggggg 2023.
[62]
OpenAl. Hello gpt-4o. http:://pppaat..cccndadeeelellep 2024.
[63] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In NeurlPS, 2022.
[64] Shraman Pramanick, Yale Song, Sayan Nag. Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. Egovipv2: Egocentric video-language pre- training with fusion in the backbone. In ICCV, 2023.
[65] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
[66] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. 2019.
[67] David Raposo, Sam Ritter, Blake Richards, Timothy Lillicrap, Peter Conway Humphreys, and Adam Santoro. Mfxture-of-depths: Dynamically allocating compute in transformer-base based language models. arXiv preprint arXiv:24044 022502258 2024.
[68] Yuzhang Shang. Mu Cai, Bingxin Xu, Yong Jac Lee, and Yan Yan. Llave-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv preprint arXiv:2403.15388 2024.
[69] Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Xun Guo, Tian Ye, Yan Lu, Jenq-Neng Hwang, et al. Moviechat: From dense token to sparse memory for long video understanding. W1X1:22307.16449, 2023.
[70] Yale Song, Eugene Byrne, Tushar Nagarajan, Huiyu Wang, Miguel Martin, and Lorenzo Torresani. Ego4d goal-step: Toward hierarchical understanding of procedural activities. In NeurIPS, 2023.
[71] Yansong Tang, Dajun Ding, Yongming Rao, Yu Zheng, Danyang Zhang, Lili Zhao, Jiwen Lu, and Jie Zhou. COIN: A large-scale dataset for comprehensive instructional video analysis. In CVPR, pages 1207- 2019.
[72] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne I Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv:2302. 122..33999 13971, 2023.
[73] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiao- qing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. 2307023077.09288 2023.
15

---

[74] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In NeurIPS, pages 6000-6010, 2017.
[75] Xiang Wang, Shiwei Zhang, Zhiwu Qing, Yuanjie Shao, Zhengrong Zuo, Changxin Gao, and Nong Sang. Oadtr: Online action detection with transformers. In ICCV, pages 7545-7555, 2021.
[76] Shiwei Wu, Joya Chen, Tong Xu, Liyi Chen, Lingfei Wu, Yao Hu, and Enhong Chen. Linking the characters: Video-oriented social graph generation via hicrarchical-cumulttie gen. In Proceedings of the 29th ACM International Conference on Multimedia, pages 47244, 2021.
[77] Shiwei Wu, Weidong He, Tong Xu, Hao Wang, and Enhong Chen. Winning the evpr"2022 '022 aqte challenge: A two-stage cenctic approach. arXiv preprint arXiv:2206. 09597, 2022.
[78] Shiwei Wu, Chao Zhang, Joya Chen, Tong Xu, Likang Wu, Yao Hu, and Enhong Chen. From a social cognitive perspective: Context-aware visual social relationship recognition. arXiv preprint arXiv:2406.00335 2024.
[79] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. WIXiv:3309.17553, 2023.
[80] Jun Xu, Tao Mci, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 5288-5296, 2016.
[81] Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-fii frree llava extension from images to videos for video dense captioning. arXiv preprint arXiv:2404.16994. 2024.
[82] Zihui Xue, Yale Song, Kristen Grauman, and Lorenzo Torresani. Egocentric video task translation. In CVPR, pages 2310-2320, 2023.
[83] Shen Yan, Xuehan Xiong, Arsha Nagrani, Anurag Amab, Zhonghao Wang, Weina Ge, David Ross, and Cordelia Schmid. Unloc: A unified framework for video localization tasks. ICCV, pages 13623- 13633, 2023.
[84] Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech. Jordi Just-,uset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. Vid2seq: Large-scale pretraining of visual language model for dense video captioning. In CVPR, pages (0714-10726 2023.
[85] Mingjia Yin, Hao Wang, Wei Guo, Yong Liu, Suojuan Zhang, Sirui Zhao, Defu Lian, and Enhong Chen. Dataset regeneration for sequential recommendation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3954-3965, 2024.
[86] Mingjia Yin, Chuhan Wu, Yufei Wang, Hao Wang, Wei Guo, Yasheng Wang, Yong Liu, Ruiming Tang, Defu Lian, and Enhong Chen. Entropy law: The story behind data compression and 11m performance. arXiv preprini N.Vv:2200.006445, 2024.
[87] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, pages 11941-11952. IEEE, 2023.
[88] Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding. 1.Xiv:2300002858 2023.
[89] Qi Zhao, Ce Zhang, Shijie Wang, Changcheng Fu, Nakul Agarwal, Kwonjoon Lee, and Chen Sun. Antgpt: Can large language models help long-ten ter- action anticipation from videos? In ICLR, 2024.
[90] Yue Zhao and Philipp Krähenbühl. Real-time online video detection with temporal smoothing transformers, In ECCV, pages 485-502, 2022.
[91] Yiwu Zhong, Licheng Yu, Yang Bai, Shangwen Li, Xueting Yan, and Yin Li. Learning procedure-aware video representation from instructional videos and their narrations. In CVPR, pages 11835,14835, 2023.
16

---

[92] Honglu Zhou, Roberto Martin-Martin, Mubbasir Kapadia, Silvio Savarese, and Juan Carlos Nicbles. Procedure-awee aware pretraining for instructional video understanding, In CVPR, pages 10727-10738, 2023.
[93] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhanc- ing vision- Lasoun-angguage understanding with advanced large language models. arXiv:230..1552, 2023.
17

---

A
Appendix
A.1
Online Video Demo
We made several demo videos to showcase VIDEOL.LM-MOD's effectiveness, available on the anonymous website
A.2 Model Architecture
We depict the overall model architecture shown in Figure 6, which is composed of three principal components: an image encoder, an MLP projector, and a language model. Following VideoI. online, we train the model with both language model loss and streaming loss.
Streaming Loss
LM Loss
Streaming
Training Target
[EOS] [EOS]
Al
[EO5]
[EOS] [EOS] Al
[e05]
Layer-N
La Language Model (Llema-33 x/c LoRA
:
Expert-i
Skipped Vision Tokens
Layer-3
Layer-2
Expert-2
Top-k Route
Selected Vision Tokens
Layer-1
User
No
ing
Tokens
Visual Tokens
MLP Projector
Image Encoder
happened the video
Describe
is washing vegetable
Al: Now, turns to cooking
Query/ Answer
Query
Frames
Frames
Figure 6: Model architecture of VIDROLLM-MOD, tokens, respectively.
represent vision, keyframe, and text
A.3 Implementation Details
For a fair comparison, we trained the models on the Ego4D narration benchmark for epochs with learning rate of 2 10-4, 4. For the Ego4D LTA benchmark, EgoExo4D fine-grained keystep recognition benchmark, and Coin benchmark, we trained the models for 6, 10, and 5 epochs with learning rates of 3 x 10 2 10-4, and 1 x 10-4, respectively. During training, size to 64 and streaming loss weight a to 1.0 by default. For the trade-off between computation cost and performance, insert LayerExpert every other layer and set the keep ratio 0.2 as the default setting.
set the batch
A.4 Vision Strategy Popular Large Multimodal Models.
We summarize the vision strategy employed in popular large multimodal models (LMMs) as shown in Table 7. The number of vision tokens increases continuously with video duration, as online video LMMs demand processing for every frame of the entire video. This leads quadratic increase in computational requirements during training and inference.
A.5 Online correction via data augmentation on temporal.
During streaming inference, we find that the model may be overfitted the previous text output, likely due to insufficient training data in the LiveChat [9] dataset. To address this problem,
18

---

Different LMMs
[TemporaSample]
#Token/frame -#Ital VTokens
BLIP-2 [39]
32 32
LLaVA [50]
256 + 256
LLaVA-1.5 [19]
576 + 576
Image-based (single image)
LLaVA-NEXT [49]
576 2880
DeepSeck- VL [52]
576 + 576
Idefics2 [35]
5 X 64 - 320
GPT-4V [60]
765 (for 1024 Res.)
Video-I.LaVA [44]
8
256 + 2048
Offline video-based (5 ~ 358)
VideoChat2 [41]
> 768
Chat-UniVi [36]
merge to 112 S-T tokens merge 356 S-T tokens
Video-ChatGPT [54]
LLaMA-VID [20]
Use all
- 1.2k (LongLoRA [14] tuned)
Online/long ideo-based (~~ 300s, e.g., 2fps)
LM-ooine [9]
Use all
1 0.66:
VIDEOLLM-MOL
Use
6k
Figure 7: Vision strategy in popular LMMs.
introduce a simple data augmentation strategy that requires no additional data annotation but, in observation, significantly alleviates overfitting. The core idea is to disrupt the language context during streaming video- language modeling, reducing the model's bias towards language alone. Our "disruption" method involves randomly shifting within temporal window and randomly replacing text with text from other timestamps within the same video data, or both. For the disrupted text, we do perform language modeling or streaming EOS modeling its adjacent video frames.
All figures presented this paper were obtained using this augmentation method. We will further investigate whether this approach can mitigate the shortage of high-quality streaming video-language data.
19

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We describe our method and contributions precisely.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We detailed our limitations in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

This work focuses on the application of video understanding and does not include theoretical results or analysis.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We clearly illustrate details and experimental settings of our approach, which rely on open-source datasets and models.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will upload our code as an attachment.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
CoodeSubmiieeeee ssi ssisssoolooooooooo oonooiiyyyyloooooooomoooomiiimii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Our work primarily uses public benchmarks for experiments, with all experi- mental settings detailed in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported due to the high computational expense involved for all settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the machine specifications and experimental details in the experi- ment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: //te::///eeeeee /ttp/nnursp.../ cc cc/ppbblcc/tthccsssuueeeeeeeeeee
Answer: [Yes]
Justification: Our work complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In the introduction and conclusion sections, we discuss how our work could be applied wel-world video applications such as an online assistant.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We do not collect data ourselves; instead, we rely on the original safeguards of the datasets and models we use.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use open-source datasets such as Ego4D, EgoExo4D, and COIN, and have obtained official licenses for them.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. cow datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide comprehensive documentation for all new assets introduced in the
paper.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We don't crowdsourcing and research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "NqBPgqcAJ3", "reviewer": "Reviewer_HwD3", "summary": "The core idea of this paper is to scaling up vision resolution for online video large language models. Instead of distributing FLOPs uniformly across all vision tokens in every decoder layer, they utilize a learnable module LayerExpert to allocate compute to critical vision tokens within the frame dynamically.", "questions": "Will you make your code available for the community?", "limitations": "The method does talk about increasing spatial resolution, but it doesn't consider spatial grounding. how does the author suggest the method can be extended for grounding?", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Video being such a compute heavy workload, anything to do with temporal modeling if can be performed in a streaming fashion can enable several practical applications. Online video-llms are relatively unexplored, this paper proposes sparse vision encoder suitable for enabling streaming applications at the same time retain spatial resolution. The project is timely, the results are good. \n\nThe results on 3 egocentric benchmarks show encouraging results.", "weaknesses": "Not a major weakness; but performing experiments on ActivityNet-based training and evaluating on ViDSTG would give an idea of how does this MoD approach for making a sparse vision encoder work on standard benchmarks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your positive comments on our work and will address each of the issues you mentioned below.\n\n**W1: Performing experiments to show how does this MoD approach for making a sparse vision encoder work on standard benchmarks.** \n\nTo validate the effectiveness and generalization of our proposed method, we conducted experiments using the same data configuration as LLaVA/LLaMA-VID on standard image and video benchmarks, as shown in Global-Rebuttal Tables 1-4.\n\nSpecifically, for video benchmarks, we trained our VideoLLM-MoD on the ActivityNet and WebVid-2.5m datasets and further evaluated it on several other benchmarks. Our method achieved comparable performance with significantly lower training costs and FLOPs, demonstrating that the proposed sparse vision processing strategy is broadly applicable to vision tasks in the LMMs field.\n\n**Q1: Code availability.** \n\nThe project code is included in the supplementary materials. We will release all the code, data, and checkpoints as soon as possible.\n\n**L1: How does the author suggest the method can be extended for grounding?** \n\nOur proposed method can process more vision tokens within the same computational budget. For spatial grounding tasks, this capability allows for larger spatial resolutions and denser frame representations. More vision tokens facilitate finer-grained representations of images and videos, capturing more detailed visual features. This is crucial for accurately locating and distinguishing small objects or intricate details within complex scenes, thereby improving the model’s precision in such scenarios.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "e1WyvlXFF8", "reviewer": "Reviewer_qExK", "summary": "This paper proposes a novel layer skipping approach to reduce the computation and memory consumption in modern vision-language models. The overall performance is good on several egocentric video understanding benchmarks.", "questions": "1. In Table 3, it is a little bit confusing why increasing the keep ratio r=0.3 results in worse performance? Also, it would be good to see the performance comparison on more keep ratios.\n\n2. Could you please explain why the proposed method targets at online video processing? It seems that LayerExpert can be adapted to a variety of vision transformers and applied to many video understanding tasks, as proved in Table 4.", "limitations": "Please refer to Weaknesses and Questions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe proposed layer skipping strategy enables efficiient attention computation while retaining the performance.\n2.\tExperimental results on mutliple online/offline benchmark datasets demonstrate the effectiveness of the method.", "weaknesses": "1.\tThis paper uses videollm-online as the baseline and merely proposes a weighted layer skipping strategy. The technical contribution of this method is relatively limited.\n2.\tThe discussion on the choice of LayerExpert is not fully discussed. The authors claimed that the proposed strategy can select critical visual tokens in the layer. It would be better if the authors include visualization results to prove this claim. In Table 3, are there any semantic-aware token selection strategies that can be used for comparison?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your thoughtful feedback and valuable suggestions.\n\n**W1: The technical contribution of this method is relatively limited.** \n\nOur technical contributions are summarized as follows:\n\n1. **Efficient Vision Modeling in Context:** It is non-trivial to reduce computation in context under online video scenarios, as online VideoLLMs must process every incoming visual token without perform frame sampling or offline token merging, as done in Chat-UniVi [1]. We are the first to explore efficient vision modeling in context, rather than merely offline pruning or merging vision tokens.\n2. **Reduction of both Training and Inference Costs:** High training costs are a significant issue for LMMs, particularly for video-based LMMs, since video consumes the majority of tokens, as indicated in Figure 7 of our paper. Unlike existing methods [2,3] that focus on efficient inference while still requiring high training costs, we are the first to successfully reduce both training and inference costs massively while maintaining LMMs’ performance. We believe our approach offers a new paradigm for other vision-language tasks, especially in training video-based LLMs.\n3. **Generalization:** To validate the effectiveness and generalization of our proposed method, we conducted experiments using the same training recipe as LLaVA/LLaMA-VID on standard image and video benchmarks, as shown in Global-Rebuttal Tables 1-4. Our method achieved comparable performance with significantly lower training costs and FLOPs, demonstrating that the proposed sparse vision processing strategy is broadly applicable to vision-language tasks in the LMMs field.\n\nThe topic of sparse vision modeling in context has gained popularity, as evidenced by Google DeepMind’s recent research on MoNE [4], released a few days ago. While they adopt a similar approach, their exploration is limited to traditional vision architectures rather than the popular LMMs field. We believe our idea \"the importance of different vision tokens should be considered in context, and can be modeled by computation budget\" can provide valuable insights into general LMMs.\n\n**W2: The discussion on the choice of LayerExpert, the visualization to prove the claim, the semantic-aware token selection strategies for comparison.** \n\nForcing LayerExpert to select only the important visual tokens across each transformer block can be viewed as encouraging LMMs to focus on the most useful regions, while it increases the learning difficulty. As shown in Global-Rebuttal Figure 1, we visualize the tokens selected by LayerExpert and observe that it indeed focuses on important visual regions, such as bike-related tokens in Fig. 1a, tokens related to slicing onions in Fig. 1b, and tokens related to a table saw in Fig. 1c.\n\nSemantic-aware token selection typically requires cross-attention-based modeling, which is computationally expensive when processing every frame. As shown in Global-Rebuttal Tables 1-2, LLaMA-VID [5] utilizes text features extracted from a q-former to select semantic-aware visual tokens via context-attention, resulting in significantly higher training costs (10.5 hours vs. 52.5 hours) with only marginal performance improvement.\n\n**Q1: Why increasing the keep ratio r=0.3 results in worse performance? Also, it would be good to see the performance comparison on more keep ratios.** \n\nFirst, selecting important visual tokens via LayerExpert increases the learning difficulty, leading to better performance with fewer visual tokens. Second, there may be deviations in the LMMs’ training process. We will conduct additional trials in the revised version and report the standard deviation across all experiments.\n\nFurther ablation studies on the keep ratio are presented below.\n\n| r    | LM-PPL | TimeDiff | Fluency | LM-Correctness |\n| ---- | ------ | -------- | ------- | -------------- |\n| 0.1  | 2.43   | 2.11     | 44.7%   | 48.1%          |\n| 0.2  | 2.41   | 2.04     | 45.2%   | 48.9%          |\n| 0.3  | 2.41   | 2.05     | 44.9%   | 48.7%          |\n| 0.5  | 2.41   | 2.03     | 45.1%   | 48.8%          |\n| 0.7  | 2.40   | 2.05     | 45.2%   | 48.9%          |\n| 0.9  | 2.40   | 2.04     | 45.3%   | 49.1%          |\n| Full | 2.40   | 2.05     | 45.3%   | 49.0%          |\n\n**Q2: Why the proposed method targets at online video processing?** \n\nProcessing video in an online scenario differs from offline processing, as LMMs must handle every incoming frame in real-time without frame sampling or token merging. This results in excessive vision token lengths, as shown in Figure 7 of the paper, and the training costs for LMMs increase exponentially with token length. Moreover, online VideoLLMs like GPT-4o have shown great potential in real-world applications, highlighting the urgent need to explore approaches that reduce both training and inference computational costs in online video scenarios.\n\nTo validate the effectiveness and generalization of our proposed method, we conducted experiments using the same training recipe as LLaVA/LLaMA-VID on standard image and video benchmarks, as shown in Global-Rebuttal Tables 1-4. Our method achieved comparable performance with significantly lower training costs and FLOPs, demonstrating that the proposed sparse vision processing strategy is broadly applicable to vision-language tasks in the LMMs field.\n\n\n[1] Liang Chen et al. An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. ECCV 2024.\n\n[2] Yuzhang Shang et al. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv: 2403.15388.\n\n[3] Peng Jin et al. Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding. CVPR 2024.\n\n[4] Gagan Jain et al. Mixture of Nested Experts: Adaptive Processing of Visual Tokens. arXiv: 2407.19985.\n\n[5] Yanwei Li et al. LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models. ECCV 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AGdRaJU5cG", "reviewer": "Reviewer_2CSB", "summary": "The document presents a novel approach called VideoLLM-MoD, which aims to efficiently scale up the vision resolution for online video large language models (VideoLLMs) without incurring high computational costs. The approach is inspired by the \"mixture-of-depths\" approach and learns to skip the computation for a high proportion of vision tokens. Experiments on several egocentric and instructional dataset show the method could achieve similar or even better performance with significantly less computation effort.", "questions": "1. could authors summarize the original ideas or insights different from Mixture-of-Depths, except of different modalities?", "limitations": "authors adequately addressed the limitations.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. the proposed method could reduce computational cost and save GPU memory, thus high vision resolution could be used for videoLLM.\n2. lightweight LayerExpert is proposed to determine which vision tokens should be processed at certain layers.\n3. streamingloss is proposed to ensure model remain silent when it has no necessary response.", "weaknesses": "1. this proposal mostly borrow idea from  \"Mixture-of-Depths\", the novel part is applying the idea to visual tokens. The original contributions may not be enough for top-tier conference.\n2. experiments are mainly on egocentric and instructional datasets, lack of  Diversity.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and efforts in reviewing our paper. We will address each of your concerns point by point.\n\n**W1/Q1: This novelty and contributions compared to Mixture-of-Depths.** \n\nWe summarize the differences between our proposed framework and Mixture-of-Depths (MoD) as follows:\n\n1. **Validation of Effectiveness on LMMs:** It is non-trivial to validate the effectiveness of the proposed approach in Large Language Models(LLMs) especially in Large Multi-modal Models(LMMs). While MoD demonstrates comparable loss scales on language models smaller than 1B parameters, we explore efficient methods to reduce computation in context for 8B-LMMs in both training and inference, particularly under online video scenarios.\n2. **Causal Operations:** While MoD introduces causality through auxiliary loss or auxiliary MLP predictors, we simplify this by using LayerExpert to select vision tokens within individual frames across transformer blocks. The causal attention in the LLM then learns the temporal modeling, seamlessly accommodating our online video scenario.\n3. **Extensive Experiments:** Unlike MoD, which validates its approach on language models smaller than 1B parameters on loss scale, we conducted extensive experiments on vision-language benchmarks, including the Ego4D narration benchmark, Ego4D LTA benchmark, EgoExo4D Fine-grained Keystep Recognition task, and COIN benchmarks. Additionally, we performed experiments on general image/video benchmarks, as shown in Global-Rebuttal Tables 1-4, demonstrating the generalization of our proposed approach to vision-language tasks.\n\nThe topic of sparse vision modeling in context has gained popularity, as evidenced by Google DeepMind’s recent research on MoNE [1], released a few days ago. While they adopt a similar approach, their exploration is limited to traditional vision architectures rather than the popular LMMs field. We believe our idea \"the importance of different vision tokens should be considered in context, and can be modeled by computation budget\" can provide valuable insights into general LMMs.\n\n**W2: Experiments are mainly on egocentric and instructional datasets, lack of Diversity.** \n\nTo validate the effectiveness and generalization of our proposed method, we conducted experiments using the same training recipe as LLaVA/LLaMA-VID on standard image and video benchmarks, as shown in Global-Rebuttal Tables1-4. Our method achieved comparable performance with significantly lower training costs and FLOPs, demonstrating that the proposed sparse vision processing strategy is broadly applicable to vision-language tasks in the LMMs field.\n\n[1] Gagan Jain et al. Mixture of Nested Experts: Adaptive Processing of Visual Tokens. arXiv: 2407.19985.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "g2uSTekDdM", "reviewer": "Reviewer_x5JZ", "summary": "This paper aims to comprehend long video streams with multi-modal large language models. The existing works use too few visual tokens to represent the video stream to guarantee efficiency but may sacrifice visual perception performance. The authors propose to keep more visual tokens to represent each video frame but only select the crucial visual tokens to pass the transformer decoder layers to reduce computation. The experiments show that the proposed mixture of depth architecture achieves a trade-off between the number of visual tokens and the computation efficiency.", "questions": "It is better to also compare with some video LLM works that compress each frame into fewer tokens, e.g., LLaMA-VID [33] in terms of both performance and efficiency.", "limitations": "The authors have discussed the limitations in the lack of experiments on exocentric data.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The explored problem is meaningful. Handling long videos with large language models is promising and has wide applications.\n2. The motivation for using different depths of networks to process visual tokens is clear and makes sense.\n3. The proposed method is simple and straightforward, and can effectively speed up the long video context processing without reducing the token number.", "weaknesses": "1. The presentation of Section 3.1 and the Figure 6 are quite similar to VideoLLM-online [8]. And in Eq.1, why are the indicator and probability terms multiplied inside the logarithm?\n2. The quantitative performance improvement over VideoLLM-online is quite marginal. And even the full computation only results in slight performance gains on some datasets. What is the underlying reason? Does it mean preserving very few visual tokens is sufficient?\n3. Some visualizations are redundant, e.g., Figure 1 and Figure 5 are very similar. There lacks the visualization on the selected visual tokens in different layers. It is necessary to show the selection results to show what is crucial in the learned LayerExpert.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for your insightful feedback and valuable suggestions.\n\n**W1: The problem of the indicator.** \n\nThanks for pointing this out. We follow VideoLLM-online and it makes an error. The revised equation is as follows:\n$$\nL = \\frac{1}{N}\\sum^N_{j=1}\\left(-l_{j+1}\\log P_j^{\\texttt{[Txt$_{j+1}$]}} - \\sigma s_j\\log P_j^{\\texttt{[EOS]}}\\right),\n$$\n\n**W2: The reason of marginal improvements over VideoLLM-online and the similar results of full computation.** \n\nThe online narration benchmark only requires generating simple time-synchronized narrations, such as “You are riding the bike,” without needing detailed descriptions that require fine-grained visual perception. Therefore, the performance improvement with increased resolution is marginal.\n\nHowever, experiments on other benchmarks, particularly those requiring fine-grained visual perception, demonstrate the necessity and benefit of increased resolution. As shown in Table 5b, for example, Ego Accuracy increased from 40.53% to 44.85% on the EgoExo4D Fine-grained Keystep Recognition benchmarks, indicating the potential of our approach for high-resolution vision tasks.\n\n**W3: Some visualizations are redundant, e.g., Figure 1 and Figure 5. There lacks the visualization on the selected visual tokens in different layers. It is necessary to show what is crucial in the learned LayerExpert.**\n\nWe will reorganize our visualizations in the revised version. We also visualized the selected vision tokens learned by LayerExpert, as shown in Global-Rebuttal Figure 1. LayerExpert effectively focuses on critical vision tokens, such as bike-related tokens in Fig. 1a, tokens related to slicing onions in Fig. 1b, and tokens related to a table saw in Fig. 1c., and the model will attend to different tokens at different layers shown in Fig. 1d.\n\n**Q1: Compare with some video LLM works that compress each frame into fewer tokens, e.g., LLaMA-VID [33] in terms of both performance and efficiency.** \n\nWe compare the performance and efficiency of our approach with LLaMA-VID [1], as shown in Global-Rebuttal Tables 1-4. Our approach achieves comparable performance to LLaMA-VID while requiring only 0.57x FLOPs and 0.2x training time.\n\n| Method       | TFLOPs | Training Cost (Pretrain + Finetune) | GQA  | MME    | POPE | SQA  |\n| ------------ | ------ | ----------------------------------- | ---- | ------ | ---- | ---- |\n| LLaMA-VID    | 9.8    | 4hrs + 48.5hrs                      | 64.3 | 1521.4 | 86.0 | 68.3 |\n| VideoLLM-MoD | 5.8    | 2hrs + 8.5hrs                       | 62.8 | 1505.5 | 85.5 | 70.2 |\n\n| Method       | TFLOPs | Training Cost (Pretrain + Finetune) | MSVD-QA |       | MSRVTT-QA |       | ActivityNet-QA |       |\n| ------------ | ------ | ----------------------------------- | ------- | ----- | --------- | ----- | -------------- | ----- |\n|              |        |                                     | Acc     | Score | Acc       | Score | Acc            | Score |\n| LLaMA-VID    | 40.1   | 9hrs + 30hrs                        | 69.7    | 3.7   | 57.7      | 3.2   | 47.4           | 3.3   |\n| VideoLLM-MoD | 23.0   | 4hrs + 5.5hrs                       | 68.5    | 3.7   | 58.2      | 3.3   | 46.3           | 3.2   |\n\nWe highlight the advantages of our proposed approach over existing efficient vision modeling methods in the LMMs field:\n\n1. **Utilizing Fewer, Semantic-Aware Tokens:** Semantic-aware token selection typically requires cross-attention-based modeling, which is computationally expensive when processing every frame. As shown in Global-Rebuttal Tables 1-2, LLaMA-VID [1] uses text features extracted from a q-former to select semantic-aware visual tokens via context-attention, resulting in significantly higher training costs (10.5 hours vs. 52.5 hours) with only marginal performance improvement.\n2. **Efficient Inference:** High training costs are a significant issue for LMMs, particularly for video-based LMMs, as video consumes the majority of tokens, as indicated in Figure 7 of our paper. Unlike existing methods [2,3] that focus on efficient inference while still requiring high training costs, we successfully reduce both training and inference costs massively while maintaining LMMs’ performance. Our approach provides a new paradigm for other vision-language tasks, especially in training video-based LLMs, and plays a role in democratizing AI by enabling broader access to trained models without the need for large resources.\n3. **Offline Spatial-Temporal Token Merging:** Online video LLMs must process every incoming visual token in real time and cannot perform frame sampling or offline token merging as Chat-UniVi [4] does. We are the first to explore efficient vision modeling in context rather than merely offline pruning or merging vision tokens.\n\n**L1: The authors have discussed the limitations in the lack of experiments on exocentric data.** \n\nIn addition to extensive experiments on ego-centric datasets, we have conducted experiments on exo-centric COIN benchmarks, as shown in Paper Table 4. To further validate the effectiveness and generalization of our proposed method, we conducted experiments using the same training recipe as LLaVA/LLaMA-VID on standard image and video benchmarks, as shown in Global-Rebuttal Tables 1-4. Our method achieved comparable performance with significantly lower training costs and FLOPs, demonstrating that the proposed sparse vision processing strategy is broadly applicable to vision-language tasks in the LMMs field.\n\n[1] Yanwei Li et al. LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models. ECCV 2024.\n\n[2] Liang Chen et al. An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models. ECCV 2024.\n\n[3] Yuzhang Shang et al. Llava-prumerge: Adaptive token reduction for efficient large multimodal models. arXiv: 2403.15388.\n\n[4] Peng Jin et al. Chat-UniVi: Unified Visual Representation Empowers Large Language Models with Image and Video Understanding. CVPR 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We would like to thank all reviewers for their constructive comments. 

We appreciate their recognition of our **motivation** (HwD3, x5JZ); the **novelty** of the approach (qExK, 2CSB); the **efficiency** (HwD3, qExK, 2CSB, x5JZ); and **sufficient experiments** (HwD3, qExK, 2CSB).

In the uploaded PDF of the Global-Rebuttal, we visualize the selected visual tokens of LayerExpert and the generated response in Figure 1. Specifically, we trained our VideoLLM-MoD on the videollm-online-chat-ego4d-134k [1] dataset. We then used five consecutive frames from the Ego4D test set videos as inputs, aggregated and normalized the vision weights from each LayerExpert, and visualized them with an alpha mask in Figure 1(a,b,c), as well as the visualization on each LayerExpert in Figure1d. Note that more transparent tokens represent larger vision weights.

We further conduct more experiments on general image/video benchmarks as shown in Tables 1-4. Our method achieved comparable performance with significantly lower training costs and FLOPs, demonstrating that the proposed sparse vision processing strategy is broadly applicable to vision-language tasks in the LMMs field.

For a fair comparison, we implemented our approach in the LLaVA codebase and trained it using the same recipe as LLaVA [2] and LLaMA-VID [3] for image and video benchmarks, respectively. We computed the FLOPs for each method using text input token lengths of 60 and one single image.

For effective training, we utilized Deepspeed zero2 and Flash-Attention2. We trained the model using LoRA with a rank of 128 and a scaling factor of 256 on all linear layers of the LLMs.

[1] Joya Chen et al. VideoLLM-online: Online Video Large Language Model for Streaming Video. CVPR 2024.

[2] Haotian Liu et al. Visual Instruction Tuning. NeurIPS 2023.

[3] Yanwei Li et al. LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models. ECCV 2024.

In the following, we address each reviewer’s concerns. For each review, we address the **W**eaknesses, **Q**uestions, and **L**imitations point by point.

Meta-review: The paper proposes a method to enhance vision resolution for online video large language models (VideoLLMs), by dynamically allocating computational resources to critical vision tokens using a learnable module called LayerExpert. The reviewers generally agree on the significance of this approach, particularly in reducing computation while retaining spatial resolution, making it suitable for streaming applications in video processing. 

There are also concerns raised about the novelty, with some similarities to existing methods, e.g. "Mixture-of-Depths" and questioning the marginal performance improvements. Further experimentation on more diverse datasets, and a deeper exploration of the LayerExpert’s decision-making process are important in the revision.
Decision: Accept (poster)