Unifying Specialized Visual Encoders for Video Language Models
Authors: Jihoon Chung
Tyler Zhu
Max Gonzalez Saez-Diez
Juan Carlos Niebles
Honglu Zhou
Olga Russakovsky
Venue: ICML 2025 poster (2025)
Keywords: video understanding
multimodal
Primary area: applications->computer_vision
TL;DR: We propose a VideoLLM approach which fuses multiple visual encoders effectively and combines their specialized knowledge into one model.

Abstract: Recent advances in vision backbones have yielded powerful and diverse visual and video encoders. Yet, current Video Large Language Models encode visual inputs using an encoder from a single backbone family, limiting the amount and type of visual information they can process. We propose MERV, a Multi-Encoder Video Representation, which utilizes multiple encoders for a comprehensive video representation. To optimize heterogeneous features from a broad spectrum of encoders and ensure efficient and coherent feature integration, MERV first aligns encoder features spatio-temporally, then projects them into a unified structure, and finally fuses them through cross-attention. Under fair comparison, MERV achieves up to 4.62% higher accuracy than its base model, while introducing minimal extra parameters and training faster than equivalent single-encoder methods after parallelizing visual processing. Qualitative analysis shows MERV successfully captures and integrates domain knowledge from each encoder, opening new possibilities for scaling enhanced video understanding.

Unifying Specialized Visual Encoders for Video Language Models
Jihoon Chung" Tyler Zhu 1 Max Gonzalez Saez-Diez
Juan Carlos Niebles? 2 Honglu Zhou? Olga Russakovsky
Abstract
SIND22 WAT
Sqy
Recent advances in vision backbones have yielded powerful and diverse visual and video encoders. Yet, current Video Large Language Models en- code visual inputs using an encoder from single backbone family, limiting the amount and type of visual information they can process. We propose MERV, a Multi-Encoder Video Representation, which utilizes multiple encoders for compre- hensive video representation. To optimize het- erogeneous features from a broad spectrum of encoders and ensure efficient and coherent fea- ture integration, MERV first aligns encoder fea- tures temporrlly then projects them into a unified structure, and finally fuses them through cross-attention. Under fair comparison, MERV achieves up 4.62% higher accuracy than its base model, while introducing minimal extra param- eters and training faster than equivalent single- encoder methods after parallelizing visual pro- cessing. Qualitative analysis shows MERV suc- cessfully captures and integrates domain knowl- edge from each encoder, opening new possibilities for scaling enhanced video understanding.
A.
C.
C.
C.
THE putio Thataiie
X
X
B.
A
A
X
X
X
A
B.
A
X
A
A
C.
C.
X
Figure I. Examples where a single encoder model the only model to correctly answer Perception Test questions (Pätrüccan et al., 2023), while MERV can correctly answer all types.
language space, allowing for reasoning to happen in the highly expressive language domain.
Most multimodal LLMs, such as LLaVA (Liu et al., 2023) for images and Video-LLaVA (Lin et al., 2024) for videos, opt for contrastively pre-trained encoders like CLIP (Rad- ford et al., 2021) and LanguageBind (Zhu et al., 2024a). Their vision- lasiunnlanguage pre-training naturally lends itself as a bridge between the vision input and the LLM, circumventing the need to train heavy vision-language alignment modules like QFommer (Li et al., 2022). These encoders al- most always pre-trained separately and vary in architecture, training data, and optimization strategy. Consequently, the features extracted by these encoders exhibit unique charac- teristics, each with inherent strengths and limitations. Con- trastive encoders like CLIP may be better suited with their multimodal semantic alignment, but inferior to models such as DINOv2 (Oquab al., 2023) fine-grained object level understanding. They also fail to take advantage of models trained specifically on videos, such as ViVIT (Amab et al., 2021). Despite this clear tension between vision back- bones, previous research in VideoLLMs has relied on only one vision encoder for visual processing as one was thought to be sufficient for visual understanding, and already dif- ficult enough to achieve vision-language alignment with. Any more encoders was unnecessary and not an effective tradeoff of runtime for compute.
1. Introduction
Inspired by the sophisticated reasoning abilities of recent Large Language Models (LLMs) (Chiang et al., 2023; Chowdhery et al., 2023; OpenAL, 2023), researchers have fo- cused on using them in many other domains to great success. The video counterparts, known Video Large Language Models (VideoL.LMs) (Bain et al., 2021; Li al., 2023c; Lin et al., 2024; Luo et al., 2023; Maaz et al., 2024; Yu et al., 2024b), connect pre- pre-trained vision encoders to LLMs by training a modality bridge from the vision space to the
Equal contribution 'Department of Computer Science, Prince- ton University, Princeton, NJ, United States 'Salesforce Research, Palo Alto, CA, United States. Correspondence to: Jihoon Chung. Tyler Zhu <jc5933,tyleran @ princctoon.edu>.
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
In this paper, we argue that this choice to not use multi- ple encoders in existing VideoLLMs unnecessarily restricts their capabilities. For example, in Figure we can see

---

Unifying Specialized Visual Encoders for Vidco Language Models
ActivityNet-QA
Enceder J
Peal
TVQA
Modell e aagee
Vissi Erceder
Feature Fusion
Pool
Three.
MSRVTT-QA
60.0
05.0 VLEP
Mesi a
Input Video
Pool
"How many pencila are insida the drawn region?"
52
M4.
Vosal Dicoder
TGIF-QA
NEXT-QA
47.9
Perception
Vioro-Car
Mece-CratoPP
LLAMAA D-78
Feature Extraction
Multi-Encader Feature Alignment and Fusion
LLM Generation
SchIA
(Dura)
Mocc-LLoVA
Figure 2. MERV architecture and performance. (Left) MERV proceeds in three main stages. First, feed in our input video into each of visual encoders to get different representations. They are then -patio-temporally aligned before being fused by a
mixer. The output is visual embedding with additive mix of information from all the encoders, which is combined with the text query generate the result. (Right) We only compare MERV to the 7B prior models trained with comparable training data mixes. MERV on on-par or better than these single-encoder prior works, while deliver considerable gains over Vidco-LLaVA (Lin al., 2024), from which MERV was initially adapted to form a Multi-Encoodrr Vidco Representation.
cases where only of four different single-encoder mod- els answers a given question correctly. While simple scene descriptions can be answered by image-level models, other questions require temporal and action-level comprehension, benefiting from features encoded with video models like ViViT (Arnab et al., 2021). Consequently, the reasoning capabilities of these LideoLLMs are directly limited by the inherent weaknesses of their respective pre-trained encoders. Therefore, employing multiple encoders could allow to complement one encoder's weaknesses with another en- coder's s strengths. It also offers a cheap way to broaden the input distribution, leveraging different encoder's training mixtures. The wide adoption of the LLaVA paradigm is also indication that vision-language alignment is simple to achieve, even without language-aware vision models.
capabilities the et al., 2017). We show that MERV can accurately capture both contrastive encoders' (Zhai et al., 2023; Zhu et al., 2024a) strengths on general vision-language understand- ing, as well as ViViT's (Arnab et al., 2021) specialty on temporally-sensitvv tasks (e.g. distinguishing pushing left vs. right), without trading off performance between these specializations as single encoder models do.
v2 dataset (Goyal
2. Related Works
VideoLLMs build upon the powerful reasoning capabil- ities of LLMs by utilizing them as language decoders to enable instruction-follower video understanding. Key advancements include VideoChat (Li al., 2023c) and Video-LLaMA (Zhang al., 2023) for chat-based video understanding. LLaMA-Adapter (Zhang et al., 2024b) for pre-aligmment, Valley (Luo et al., 2023) with multilingual L.Mss, Intern Video (Wang et al., 2022) with a dedicated video encoder training phase, and Video- ChotChh (Maaz et al., 2024) combining video-adapted encoders with LLMs. GPT4Video (Wang al., 2023) supports video understand- ing and generation, while MovieChat (Song et al., 2024) focuses on long video comprehension. Models like Chat- UniVi (Jin et al., 2024), LLaMA-VID (Li et al., 2024b), and VTM (Lee et al., 2024) optimize token usage for video representation. Other notable models include Vamos (Wang et al., 2024b), which flexibly uses visual embeddings, ac- tion labels, and video captions as input; VideoChat2 (Li et al., 2024a), developed through three-stage progressive training; Video-LLaVA (Lin et al., 2024), which aligns im- age and video representations before projecting them to the
We propose MERV, Multi-Encoder Representation of Videos, as a new method for integrating multiple visual encoders into a single VideoLLM using a cross-attentive encoder mixer for fusing representations. We introduce a spatio-temporally aligned representation for mixing the in- formation from multiple types of visual encoders. Given the computational complexity of video tasks, we carefully experiment with optimization strategies and parallelizing the visual experts, allowing to combine four distinct vi- sual encoders with minimal computational overhead. Our frozen method outperforms all individual encoder methods, up to 4.6% better than prior works (Lin al., 2024) on video reasoning benchmarks, i.e., from 37.66% to 42.28% on TVQA (Lei et al., 2018), and on par with SeViLA (Yu al., 2024b) Perception Test (Päträucean et al., 2023), challenging perception and reasoning diagnostic for video models. Finetuning the full model improves MERV past SeViLA (Yu et al., 2024b) by 2.2%, from 46.2% to 48.4%. Finally, we do a detailed qualitative study of our model's
'Our code and pretrained weights available at ht https: //github. com/ wwtp/oooooooeeecccce wwwpprinnnnniiieeccomoooooiiiiie www.wvssslllaamaaiaama al/eerv.
2

---

Unitying Specialized Visual Encoders for Vidco Language Models
LLM space; and VideoPrism (Zhao et al., 2024), which also further trains a video encoder through masked distillation. Specialized models like VTimeLLM (Huang et al., 2024) focus on fine- the-graaded video moment understanding and time- bound reasoning, while models like Elysium (Wang et al., 2024a) and Merlin (Yu et al., 2024a) can predict object tra- jectories. SeViLA (Yu et al., 2024b) uses LLM for frame localizer of the video for multiple-choie tasks, Finally, re- cently LLa VA-Hound-DPO (Zhang et al., 2024a) explored using DPO and a higher quality training set for better instruc- tion following. Distinct from these aforementioned works, our approach centers on utilizing a diverse array of visual and video encoders, which are RGBB-asee but coming from different visual backbone families, each with its own unique strengths, significantly enhance the capabilities of the VideoLLM framework. By strategically using these spe- cialized encoders, we aim to capture broader spectrum of visual information, enriching VideoL.LMs' understanding of videos.
3. MERV: Multi-Encoder Representation of Videos
Our goal for MERV is to systematically build video model that leverages multiple encoders with an LLM to process a video following the LLaVA/PrefiXLM (Liu et al., 2023; 2018) paradigm (see Figure 2). Unlike previous works, our focus is not on combining multiple encoders of different modalities that make use of additional information (depth, audio, etc) (Bachmann al., 2022; Zhu et al., 2024a), but instead focus on RGB-encoders trained on different datasets and objectives that offer different visual understanding. We extensively ablate three key aspects to make this possi- ble: our selection of multiple encoders, i.e., which visual encoders and how many to use (Sec 3.1); how we align the spatio-temporal representations of each encoder to mix the information together, especially in an efficient manner (Sec 3.2); and our implementation efficiencies, from the parallel visual processing to the training recipes (Sec 3.3).
Combining multiple encoders for multimodal LLMs is gaining attention. Eyes Wide Shut (Tong et al., 2024b) explored mixing DINOv2 and CLIP features for LLaVA, but their results signal that mixing features effectively re- quires investigation. Both Mipha (Zhu et al., 2024b) and Prismatic- VLMs (Karamcheti et al., 2024) found that image encoders like CLIP and SigLIP, which are trained using vision-language contrastive loss, surpass other image en- coders such as ViT and DINOv2, with SigLIP showing further improvements over CLIP. SPHINX-X (Gao et al., 2024) and SPHINX (Lin et al., 2023) combines multiple image encoders by concatenating features along the chan- nel dimension, while BRAVE (Kar et al., 2024) concate- nates features from multiple encoders sequence-wise, fol- lowed by a QFormer with masked modeling. More recently, Cambrian-1 (Tong et al., 2024a) creates a multimodal LLM which spatially aligns their inputs across different resolu- tions. This also reproduces our finding of the importance of spatio-temporal alignment. There is also the popular body of research on multimodal LLMs using many modali- ties including image, vidco, audio and/or 3D (Chen et al., 2023; Han et al., 2023; Li et al., 2023a; Lyu et al., 2023; Panagopoulou et al., 2023; Su et al., 2023; Sun et al., 2023a; Zhang et al., 2023; Liu et al., 2024a; Han et al., 2024; Liu et al., 2024b; Jain et al., 2024). In contrast, this paper dives into the video-language domain, exploring combining multi- ple image and video encoders and exploiting their structural similarities. Our multi-encoder feature alignment and fu- sion are both performant and efficient in FLOPs, and results in an .compascng additive mixture of features which previous works could not achieve.
3.1. Multi-Encoder Feature Extraction
Our final architecture uses four distinct types of models: spa- tial experts, fine-grained temporal experts, image-language experts, and video-language experts. We found experimen- tally that our choice of four performed the best across all types of questions, and ablate our choices in Sec. 4.2. More details about these four encoders are in Appendix Table
DINOv2 (Oquab et al., 2023) is trained using unsupervised learning local-to- -ocablto-llobll correspon- dences in image data. The resulting features have robust object part understanding, as well as semantic image under- standing, but can suffer from poor language grounding.
Spatial expert:
Temporal expert: ViVIT (Arnab et al., 2021) is trained using supervised learning on short videos. The architecture is designed for modeling the interactions between frames using spatial and temporal attention, which lets it capture longer temporal dependencies than pure image models can.
Languaa contrastive expert: SigLIP (Zhai et al., 2023) is trained using sigmoid contrastive learning on image- text pairs. The model is designed to learn joint embed- ding space for images and text, which makes it good at understanding vision-language associations. However, it can overlook the finer details of an image which are not well described by text in its training data.
Video-Language contrastive expert: Finally, our video- language expert is LanguageBind (Zhu et al., 2024a). Used by Video-LLaVA (Lin et al., 2024), LanguageBind is trained through joint multimodal learning between text and mul- tiple modalities (videos, infrared, etc..) and understands the relationship between video and text and their high-level semantics. We only use the video encoder of LanguageBind.
3

---

Unifying Specialized Visual Encoders for Vidco Language Models
3.2. apatio-Temporan Aligned Representations for Feature Fusion
attention with learnable queries to additively mix the dif- ferent representations together. The visual features deter- mine the weights of the linear mixture, which we find suf- ficient for our task. We use a single randomly initialized query Q E R keys as X = [X] xv] E R R where X E I is each encoder's features (after Eq. 1) averaged over the sequence dimension for faster com- putation, and N the number of encoders, and values as X [x] xx] E R We calculate our final unified feature as
Our input is batch of text, image-text, or video-text queries. The visual part of the imput, either images vidcos, is passed through each of the visual encoders to extract the respective features. Here we describe the detailed care we took in pre-processing to prepare the features for alignment.
First, images are treated as videos with repeated frames, so assume all inputs are videos from here on out. A video is of shape T x H x W, where T is the number of frames and H, W are the height and width of the frames, and produce an output of shape te x he X We for an encoder e. One obstacle with using different visual encoders is that each model outputs features with a different structure. For exam- ple, given an input of shape 16 x 224 x 224, ViViT outputs feature of shape 8 x 14 14 whereas LanguageBind s features of shape 16 x 16 x 16. Image-based encoders will not change the temporal dimension, whereas ViViT downsamples the frames by a factor 2.
QX' T vd
0 == Softmax
X E Rexd
(2)
The final step is to concatenate the visual embedding and tokenized text together into the LLM. We use the base LLaMA-2 7B model (Touvron et al., 2023b), which we found performs better than the chat model. We test multiple alternate feature fusion strategies in Section 4.2.2.
For temporal alignment, as each encoder is flexible enough to handle varying input frames, we simply choose our input for each encoder so that each output Lg is the same across all encoders, i.c.
3.3. Implementation Efficiencies
Parallelized visual encoding. At a first glance, using multi- ple encoders seems to be a large cost to pay when comparing the raw FLOPPs and parameters. However, a key benefit of the LLaVA style architecture is that the entire feature ex- traction and projection pipeline can happen in parallel. To make this possible, we build on top of recent powerful ad- vances in parallel processing for LLMs and use Py PyTorch's Fully Sharded Data Parallel (Zhao et al., 2023). As the video encoders themselves are much smaller than the LLM blocks, and their visual encoding processes are completed in roughly the same amount of time, most of the overhead in running four encoders is already covered by having just one encoder (ref. Figure 3). We provide some timing numbers in Section 4.2.3 and find that our step time is similar to that of the single-encoder methods.
Pre-fusion projection. Now need to achieve spatial alignment among the features. Naïvely combining them will not work as they all have different spatial shapes. Us- ing the full resolution features would also be prohibitively expensive. We design a pre-fusion projector to tackle both issues by aligning and compressing the features.
Suppose our feature from encoder e is v. E Rxxh, xd. where de is the dimension of encoder e, and assume the output spatial representations are square (i.e. h we, but we keep notation for clarity). Our pre-fusion projector uses an adaptive 2D average pool P for each encoder to resize the spatial dimensions to the same h x w for all encoders, where h < he and w < We. As is the same across each Ve: this spatio-temporally aligns the representations.
Our code is built on top of the Prismatic VLM code- base (Karamcheti et al., 2024), which efficiently implements vision-language model (VLM) training. We add the ability to handle videos and an arbitrary number of visual encoders, along with many useful features for training. Our training is efficient for using multiple visual models, completing in un- der 24 hours using 8 L40-48GB GPUs, and down to 8 hours using 8 H100s. In contrast, the Video-LLaVa codebase runs the Stage-2 training in around 38 hours on the same L40 setup and could not easily support multiple encoders in our initial attempts.
Finally, we need to connect the varying embedding dimen- sions de to same dimensional space. We add linear layer to project the features from dimension de to d, the LLM's dimension. In total, our pre-fusion projection is
X. : P(V))W. E exd for e E Encoders (1)
where We E Rdd is each encoder's output linear layer, and = txhxxx w. This projector is lightweight, having only d x Σ. de trainable parameters for dimension matching, making it easy to scale to an arbitrary number of visual encoders. For detailed ablations, see Section 4.2.1.
MERV frozen and full. Many different recommendations for training LLaVA style models have been made since its inception. This is only made more complicated by the introduction of new datasets with every new VideoLLM architecture, making it difficult to properly determine the
Feature fusion strategies. The final part of our pipeline is fusing the multi-encoder information together using cross-
4

---

Unitying Specialized Visual Encoders for Vidco Language Models
Table Comparison different multimodal LLMs on video reasoning benchmarks. employ ChatGPT to evaluate performance
following
where applicable (version gpt-3. 3-5...5555 3-turbo-0613) denotes our evaluation of using the author provided
checkpoint. The first five datasets were used as evaluation test sets during development; the last three were held-out for our final evaluation.
MSVD-QA
MSRVTT-QA
TA
Perception Acc
NetivityNet-QA
NEXTQQA
VLEP
TVQA
Methods
Acc
Score
Acc
Score
Score
Acc
Scure
Acc
Acc
Acc
Alternative data mixes
Video Video-Chat (Li al., 2023c)
56.3
2.8
45.0
2.5
26.5
2.2
M.MMA-Adapeer (Zhang al., 2024h)
54.9
3.1
43.8
2.7
34.2
2.7
Video-LLaMA (Zhang al., 2023)
51.6
2.5
29.6
1.8
12.4
Video-ChaIGPT (Mauz et al., 2024)
64.9
3.3
49.3
2.8
35.2
2.7
SeViLA (Yu et al., 2024b)
46.2
63.6
64.4
38.2
LLaMA-VID-7B (Li et al., 2024b)
69.30
3.74
57.84
3.24
51.31
3.26
41.64
46.45
3.22
60.61
57.65
37.43
LLaMA-VID-13B (Li et al., 2024b)
70.25
3.77
58.58
3.26
51.26
3.26
41.54
46.79
3.23
60.03
61.98
41.33
Same data mixes
Video-LLaVA+ (Lin et al., 2024)
67.74
3.69
56,90
3.18
47.99
3.17
44.22
47.08
3.27
59.61
61.21
37.66
MERV (frozen) MERV
70.97
3.76
59.03
3.25
51.1
3.26
46.21
50.87
3.34
63.09
58.66
42.28
Gains to Video-LLaVA*
+3.23
+.07
+2.13
+.07
+3.11
4.09
+1.99
+3.79
+.07
+3.48
-2.55
44.62
MERV (full)
70.48
3.79
57.25
3.24
51.39
3.28
48.41
49.93
3.33
61.36
60.07
39.42
Gains to Vidco-LLavA
+2.74
+.10
+0.35
+.06
+3.40
+.11
+4.19
+2.85
+.06
+1.75
-1.14
+1.76
best recipe for one's own setup. We intentionally fix our dataset to be the same as Video-LLaVA's so we can iso- late the impacts of the training setup, from which we find two viable settings: MERV (frozen) and MERV (full). The original Video-LLa recipe has two stages: Stage pre-training on captioning data to align only the connector between the pre-trained vision encoder and LLM, and a Stage 2 instruction tuning on both connector and the LLM.
instructions from Video-ChatGPT (Maaz et al., 2024).
All the preprocessing, including frame extraction, adheres to the original method that each encoder is trained with. We extract 16 uniformly sampled frames from each video, except for ViVIT which extracts 32 frames by default but produces a 16-frame output feature.
For MERV (frozen), we train on only Stage 2 data for I epoch with a learning rate of 2 x 10 and batch size of 128 with gradient accumulation. For MERV (full), we first train on Stage I data with a learning rate of 1 x 10-4 and the projectors, feature fusion, and LLM unfrozen with similar settings. Both recipes use an initial warmup ratio of 0.03 and a cosine schedule.
MERV (frozen), which performs only the Stage 2 in- struction tuning inspired by Karamcheti et al. (2024); it achieves similar results to the original Video-LLaVA recipe in only 43% of the time;
MERV (full), which undergoes both of the Stage 1 cap- tioning pre-training and the Stage 2 instruction tuning,
but also unfreezes the LLM during Stage for a slight improvement few benchmarks.
Evaluation. We evaluate our model on a compre- hensive suite of video understanding benchmarks, in- cluding the open-ended MSVD-QA (Xu et al., 2017), MSRVTT-QA (Xu et al., 2017), TGIF (Jang et al., 2017), and ActivityNet-QA (Yu et al., 2019), as well as the multiple-choice benchmarks NEXT-QA (Xiao et al., 2021), VI_EP (Lei et al., 2020), TVQA (Lei et al., 2018), and Per- ception Test (Pärräccean et al., 2023). We emphasize that NExT-QA, VLEP, and TVQA datasets are held-out datasets that did not use during our experiments, and only evalu- ated once after all the design is completed. We report both accuracy and score following the Video-ChatGPT evalua- tion protocol where applicable, and all evaluations are done zero-shot without any dataset-specifie fine-tuning. Results using GPT-3.5-turbo for evaluation are done with the June 13th, 2023 cutoff date.
As MERV (frozen) is faster to train with similar perfor- mance, we adopt that recipe by default for analysis, and interchangeably use MERV to refer to it for simplicity from here on out. Detailed analysis is provided in Section 4.2.3.
4. Experimental Results
In this section, we show that our method outperforms prior works across standard lideo-language benchmarks before moving onto an in-depth analysis of our method and its specializations in the next section.
Datasets and training procedure details. For fair com- parison, our data mix is the same as Video-LLaVA (Lin et al., 2024). The Stage 1 data is single-tun concise cap- tioning, with 558k (image, text) pairs from LAION filtered by LLaVA (Liu et al., 2023) and 702k (video, text) pairs from Valley (Luo et al., 2023). The Stage data is multi- turn conversations, detailed captioning and reasoning. with 665k (image, text) pairs from LLaVA and 100k (video, text)
4.1. Comparison to State of the Art
Table 1 tabulates the performance of MERV (frozen) and (full). We compare our model to the existing works, in- cluding Video-LLaVA (Lin et al., 2024) that share our train-
5

---

Unifying Specialized Visual Encoders for Vidco Language Models
Table 2. Ablating design choices. We highlight our defaults in orange and bold the best results. Average accuracy is on MSVD, MSRVTT, TGIF, and Perception Test. Full metric results are in the Appendix.
(a)
Pre-fusion projectors. * is 16 (b) Pre-fusion output token. We ablate
(c) Feature fusion strategy. Cross-Attm the optimal token size per frame for the additive mixing is the best overall among all the strategies on accuracy, for its FLOPs.
frames instead of 8. Top two rows are projector-free baselines.
pre-fusion projector
Projector|
Avg Ace
Params FLOPs
Tkns
MSVD
MSRVTT
TGIF
Strategy
Avg Acc
FLOPs
257 lok
54.76
1
61.94
54.64
41.41
class tok
52,05
4
64.47
55.72
45.32
Cross-Attn
56.83
17.19 T
2D Avg
54.96
0
2.IM
16
67.23
56,44
47.75
Concat (Seq.)
54.45
43.09
2D Avg"
55,86
0
4.2M
64
69.08
58.00
50.01
Concat (Ch.)
56.64
16,29 T
2D Aun
52.12
12.7M
9.7G
Learnable W
55.01
16.24
100
68.38
57.47
48.78
2D Conv
3D Avg*
55,09
54.23
237M
0
241G
4.2M
144
68.65
57.73
48.81
25% - Mixed
54.19
16,39 T
3D Conv
55,42
113M
232G
256
68.46
57.72
48.66
ing data mixture, and other VideoLLMs (Li et al., 2023c; 2024b; Maaz et al., 2024; Yu et al., 2024b; Zhang et al., 2023; 2024b). We find that our method, generating video representations using multiple visual encoders that special- ize in different skills of video understanding, outperforms Video-LLaaA VA across nearly all of the benchmarks, with a 4.1 % gain on Perception Test, a 3.7% gain on ActivityNet and a 4.6% gain on TVQA. Both of our methods perform better overall than Video-LLa VA, even when using less data with just Stage 2 as shown by the MERV (frozen) numbers. While MERV (full) is not a striet improvement MERV, it still improves on some difficult benchmarks (e.g., Per- ception Test) with its additional video-language alignment. We believe that this makes MERV (full) generalize better in unseen settings outside of these testing benchmarks, and recommend using this recipe when possible. Compared to LLaMA-VID-7B, which uses a different training mix, we are better in nearly all benchmarks, up around 4.5% across Perception Test, ActivityNet, and TVQA. Moreover, MERV (full) outperforms SeViLA on the Perception Test zero-shot with 48.4%, compared to 46.2%. Overall, our design shows a significant improvement over Video-LLaVA and prior methods as a whole.
Pre-fusion projector. Table 2a tabulates our projector abla- tion on average accuracy, parameter count, and FLOPs, with the default settings of LanguageBind as the single vision encoder and 8 frame 64 token projection output. We find that 2D average pooling is the best, with 55.86% average accuracy, even better than using no projector on the full 257 token representation (as in Video-LLaVA (Lin et al., 2024)), with 54.76% average accuracy. It also has no trainable pa- rameters and the fewest FLOPs. The projection serves as a form of feature selection, allowing the LLM to efficiently reason only about the most relevant information.
One result worth noting is the poor performance (52. 3.7% lower) of the attentive resampler, a popular projector choice. The attentive resampler is less responsive to the in- put spatio-temporal structure compared to other projectors, which leads it to being a weaker projector for us. However, it would likely be better with higher quality data from which the transformer can learn. Increasing the frame resolution from 8 to 16 was also large improvement, showing that increasing temporal resolution is still important. This high- lights the importance of aligning representations with their spatial and temporal structure, especially for video models, which extract many more frames of visual information.
4.2. Ablations
Projector token length. Similarly, we ablate the output token length of the projector. Table 2b tabulates the per- formance of producing 1 to 256 tokens for 2D average pooling projector with 16 frames as input. We see that the performance peaks at 64 tokens with 69.08% MSVD ac- curacy, with worse performance for longer token lengths (max. 68.65%). This balances the number of tokens used for condensing the visual embedding while also minimizing the extra processing needed by the LLM.
In this section, we justify our design choices for the pro- jectors, feature fusion strategies, and training recipes. Our ablations are done with the MERV (frozen) recipe.
4.2.1. PRE-FUSION PROJECTORS
The first module we investigate is our projectors, which serve to connect each encoder from its pre-trained embed- ding space to a common embedding space. We test two types of projectors: image-level, which operate on frames independently, and video-level, which aggregate informa- tion across frames. Projector details are provided in Sec- tion A.3 in the Appendix. We report average performance across our development sets of MSVD, MSRVTT, TGIF, and Perception Test.
4.2.2. FEATURE FUSION STRATEGIES
Next, we test different strategies for fusing the informa- tion from all of the features, with detailed breakdowns in Table 2c. First, we evaluate two popular concatenation meth- ods, in either the token sequence dimension, or the channel
6

---

Unifying Specialized Visual Encoders for Vidco Language Models
dimension followed by an MLP for matching the LLM di- mension. While sequence-wise concatenation is widely used in multimodal LLMs (Tong et al., 2024b), our method outperforms it while using significantly less computation, with a 56.8% average accuracy compared to 54.4%, while also using 2.5x fewer FLOPs. Concatenation channel-wise reaches a similar performance of 56.6% and a lightweight cost. However, our cross- coss-attention shows better average performance, better on par on 3 out of 4 benchmarks (ref. Appendix Table 6c), with the additional benefit of having accessible encoder weightings for analysis, so we stick with cross-attention for our final design. We also try different methods of additive mixing as an ablation. The last two rows of Table 2c show the performance when either learn- ing the additive weights as a single scalar or by fixing the weights be 0.25 for each of the 4 encoders. We see that using cross attention outperforms both methods by 1.8% and 2.6%, as our feature fusion module can dynamically generate better fused embeddings based on the visual input.
Single encoder with Videol L VA codebase -....ew.........................
Average Step Time (s)
20
16.31
14.82
14.95
13,60
10
5
Additional Encoder Overhead
Slowest Single Encoder VidLM
2
3
Number of Encoders
Figure 3. Extra encoders incur minimal step time overhead. We add encoders in the order of DINOv2, LanguageBind, SigLIP, ViViT, plotted alongside the slowest single encoder in each group.
portant for optimal performance.
Finally, we provide evidence for the efficiency of our method (ref. Figure 3). We use the default FSDP shard- ing strategy Py PyTorch provides; it is not currently possible to specify explicit plans for which modules go where (but may be possible as FSDP matures). However, even with this basic strategy, method is dominated by the slowest single encoder present, incurring very little additional over- head from extra encoders due to this parallelization, making it cost-efficient to scale up in the number of encoders.
4.2.3. TRAINING RECIPES
Finally, we also compare different training recipes based on the literature and our own expertise. Apart from training recipes mentioned in Section 3.3, many recent works have attempted some combination of other strategies, such as unfreezing the vision encoders in Diao et al. (2024), which usually requires more training data.
4.3. Ensemble Composition
We systematically map out this landscape, fixing our dataset to be the same as Video- Va''s. Unlike them, we found that the Stage 1 phase did not help much when training only the projectors and feature fusion (ref. Appendix Table 5), with roughly the same average accuracy. Stage 2 instruction tuning alone leads to similar results in 43% of the total time, so we adopt this recipe for efficiency and refer to it as MERV (frozen). This recipe is still unsatisfying as it leaves a large amount of data, approximately 1.3M vision- text pairs, unused for training. In our empirical observations, we found that the resulting video-language alignment was suboptimal. The distributions of language used in video training datasets and benchmarks sparsely overlap based on their sentence embeddings, which could be impacting our ability to generalize zero-shot on downstream benchmarks. We address this by unfreezing the LLM during Stage I to better learn this alignment, improving performance on a few key benchmarks, especially Perception Test, by up to 2.2%. We call this recipe MERV (full).
The original motivation of our work was to choose encoders with complementary visual knowledge to form a comprehen- sive representation for our final model. The key questions are 1) do we benefit by using more than one encoder, and 2) do we need all four encoders, i.e. does each one meaning- fully contribute to the final performance?
Can we make use of more encoders? wisdom is to use a single encoder, typically contrastively trained vision-language model like CLIP, SigLIP, or Lan- guageBind (Radford et al., 2021; Zhai et al., 2023; Zhu et al., 2024a), in a VideoLLM. In Figure 4a (0) we show the four single encoder models corresponding each of our chosen encoders using their full embeddings. They not only all perform worse than MERV but also use more FLOPs, as without our pre-fusion projectors, their sequence lengths are at least 4x ours.
The conventional
Are each of the encoders contributing? To affirm that this set of four encoders is actually beneficial for improving understanding, we train three-encoder VideoLLMs under the same strategy, but removing a different encoder each time. Each of these models does worse based on the strength of the encoder removed, meaning that MERV is using their knowledge (ref. Fig. 4a, A). The minor drop in FLOPs illustrates how most of the computation is still dominated by the L.M,, not the vision encoders.
As another ablation, we train MERV on a single stage com- prised of the Stage and Stage 2 data mixed together (ref. bottom of Appendix Table 5). Surprisingly, this does worse than the explicit two stage training recipe. We attribute this to the explicit types of data in each stage being a form curriculum learning, showing that these stages are still im-
7

---

Unifying Specialized Visual Encoders for Vidco Language Models
LanguageBind
BEE
45 50%
56.05-
1 Sigh
MERY (idlm
Ang Accuracy
54 54.0%
SigliF
Lngupdiind
Full Datuset
-LMA--aa--B Bin
D LaMA-Vad-ISB
35 1%.0%
SigL.P
32.0%
DINOvle
DINOv2
DINOV2
30.0%
43.0%
MERR fie
I-tac VLLM
25
Stage Stage only
0.0
20111
20.0%
25.0%
50.0% Temporal
35.0%
40.0%
HLOPs
ViVIT
(a)
(b)
Figure 4. Analysis plots supporting our design of multiple en- coders, from their accuracy their skill specializations. (a) Visual Encoder Subsets. MERV outperforms single-encoder VideoLLMs (0), with our feature projectors unlocking more computational efficiency. Removing any encoder also reduces MERV performance (A), Average accuracy is across MSVD, MSRVTT, TGIF, and Perception Test. (b) SSv2-MCQ and Tem- poral. Temporal denotes performance on 12 selected classes where actions indistinguishable if played in reverse (ref. Sec. 5.3). Full results are in the Appendix Tables 7, 8, 10.
SigLIP
Figure 5. Videos that give the highest attention weight for each of the encoders. The right-most column shows the average frame of the video. For more examples. see Figure 11 the Appendix.
5. Analysis
5.1. Cross Attention Weights Activate on Corresponding Videos
best in all tasks. LanguageBind, for example, performs the best in TGIF-What with 46.23%, while DINOv2 per- forms on par with the best in MSVD-who with 82. 82.12%. Our method combines different encoders into an unified representation and consistently matches or improves the hest-performing single-encoder model. Raw numbers are in Table 9 in the Appendix.
We first understand our method by looking at the cross at- tention weights on our benchmark datasets (MSRVTT, TGIF, MSVD, and Perception Test), and visualizing the videos which have the highest attention weight for each of the encoders in Figure 5. This lets us see what types of videos activate each encoder the most. As expected, ViViT attention weights are highest on videos with large motion, as ViVIT has strong temporal understanding. Mean- while, SigLIP is utilized for videos that have textual data in the video, likely due being vision-language contrastively trained, especially with textual data during training. DI- NOv2 and LanguageBind are both preferred by videos with static scenes, but LanguageBind is preferred for videos with foreground motion.
5.3. MERV Can Intuit Motion and General Understanding Simultancously
We take another angle to quantifying how well our model learns from each of its individual encoders by looking back to classic video action recognition datasets. They are com- monly used for general action understanding, but they can also be used to evaluate finer- uner-grained capabilities. We are most interested in both general understanding and distin- guishing actions which are temporally ambiguous, i.e., indis- tinguishable when reversed in time, such Pulling [some- thing] from left to right and Pulling [something] from right to left. This offers a fair analysis of both general and finer- grained video understanding.
5.2. MERV Can Capture Visual Skills of Different Encoders
Next, we ask if our model effectively captures knowledge from its encoders. We first answer through our previous open-ended QA benchmarks. To assess the performance across different visual tasks, we create "pseudo" cate- gories by looking at the first word of the question sentence, which are often WH-words. They can be viewed proxy of skills required to solve the task. For example, Where requires spatial understanding and When requires temporal understanding. Figure 6 shows the relative performance of different visual encoders. While the contrastive models gen- erally dominate each category, no single encoder performs
We turn to the Something- (SSv2) dataset, where the goal is to classify an input video into one of 174 classes, e.g., Pulling [something] from left right. This allows us analyze of temporal-spatic interaction with minimal distractions from scene understanding and real-world semantics. How- ever, evaluating SSv2 as a zero-shot open-ended task is difficult with a long tail of specific categories. Thus we reformat the dataset into 5-way multiple-choice questions
v2 (Goyal et al., 2017)
model's understanding
8

---

Unitying Specialized Visual Encoders for Vidco Language Models
MSVD
TGIF
Relative Performace
Performinnc
LB
Relatre
DINO<?
VIVIT Sig MERV
What
Who
Hlow
When
What
Who
What
How
Where
Paching tnomsthnng from rihtt rigttsich woiat Pashing (ransbhing] en slgiddy mores
Publing fronsthing fory left right
Yaking fona muny studer the rutle)
Phosgoog (sumectiong) iso /sarsdhing Covetnng (sunatiag) with cenneching)
VIVIT
DINOV2
LangangeBiod
SigLIP
MERV
Figure 6. Single encoder vs. MERV on different types of video tasks. We plot the relative performance of VideoLLMs with dif- ferent visual encoders. While each single encoder has its strength in different tasks, our method shows better performance than all the other single encoders in almost every task. We only plot tasks with more than 500 samples. See Appendix details.
Figure 7. Single-Encoder Performance Difference in Something-Something v2 MCQ. ViVIT shows better performance tasks where temporal understanding is crucial, while LanguageBind (LB) and SigLIP show better performance where task can be solved from single-frame understanding.
(MCQ) and fix the prompt to be "How is the object in the video being interacted with?". Incorrect choices were ran- domly sampled from the other 173 classes. We call this benchmark "SSv2 MCQ" to distinguish it from the origi- nal classification task. Additionally, we selected 12 classes a priori from SSv2, where the action indistinguishable if reversed in time, e.g., Pulling [something] from left to right and Pulling [something] from right to left to form subset that can test the model's 's ability in temporal understanding.
semantics of the scene can hint the action that is happening, e.g., if a frame where a blanket is dropped on an object is given, one can easily expect Covering [something] with [something/ without watching the full video. See Appendix Figure 9 for sample videos of the these classes
6. Conclusion
Previous VideoI.LMs have been limited to relying on a single visual model for feature extraction, which leads to limited understanding capabilities of vastly different video tasks. In our work, we break this paradigm and explore various fusion strategies for combining information from multiple visual experts to generate representation that can leverage the capabilities of different video encoders. We find that our multi-encoder feature fusion is able to outperform comparable methods by up to 4.62% on video reasoning benchmarks. We show that the method can obtain better performance than the best-performing single-encoder model with minimal computational overhead. Finally, we quantita- tively and qualitatively observe the skill specializations model learns on an MCQ format of Something-Somelle v2, which confirms both that encoders can be specialized and that our model captures both axes of knowledge. Our paper proposes some initial steps in rethinking how we ap- proach the use of multiple encoders. We hope that this inspires others to also consider this problem another di- rection for scaling and improving their VideoLLMs.
Figure 4b plots the performance of MERV and single- encoder models on this temporal subset (æ-axis) against the full dataset (y-axis). We see that the ViVIT single-encoder model, which often falls short in other video QA bench- marks, surprisingly performs better than other encoders at 39.77% on temporal subset, which is 9. 1.1% higher than the next closest model LanguageBind. However for the full SSv2 - MCQ, ViViT suffers with a worse performance of 26. 36.78%,44 as ViViT's strength is on temporal understand- ing despite lacking in lision-language understanding. Con- trastive encoders have the upper-hand in most other classes.
We believe that the architecture, datasets, and objective of each model causes these difference. ViVIT processes spatial- temporal tubelets for embeddings. leading to better temporal understanding despite only being pre-trained on Kinetics- 400 classification. SigLIP uses image-based ViT with no temporal layer has limited temporal understanding, but has greater knowledge due to its larger training and con- trastive objective. MERV, 42%, shows better performance compared to all these single-encoder models via leveraging strength of all the individual encoders. MERV (full) per- forms better than both Videol.LaVA (Lin et al., 2024) and the 7B and 13B variants of LLaMA- Vid (Li et al., 2024b).
Acknowledgements
This work is supported by the National Science Foundation under Grant No. 2107048 and the Princeton First Year Fellowship to TZ. We also thank Allison Chen and William Yang for detailed comments and feedback.
Finally, we also plot the performances of six SSv2 classes where the performance difference between ViViT and SigLIP is largest in Figure 7. We that actions which cannot be inferred from single frame are the ones that ViViT performs better, c.g., Pushing [something] from left to right is indistinguishable from Pushing [something] from right to left if temporal information is omitted. Meanwhile, IPP performs better for classes where understanding the
Impact Statement
Our work aims to facilitate video understanding, which can lead to positive social impacts such as a video captioning model for fow-vision users, automatic detection of medical

---

Unitying Specialized Visual Encoders for Vidco Language Models
emergencies, better self-driving cars. It can also lead to negative social impacts like easy surveillance by the author- ities, and human-like internet bots being used for scamming purposes. We follow the same safeguards implemented by the original authors of the datasets, the visual models, and the LLM models. We have not put any additional safeguards ourselves.
contributors, S. Stablelm: Stability ai language
models, 2023. URL https://githube Stability AI/StableM..
Diao, H., Cui, Y., Li, X., Wang, Y., Lu, H., and Wang. X. Unveiling encoder- free vision-language models. NeurlPS, 2024.
Fang, Y., Wang. W. Xie, B., Sun, Q. Wu, L., Wang, X., Huang, T., Wang, X., and Cao, Y. Eva: Exploring the limits of masked visual representation learning at scale. CVPR, 2023.
References
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et al. Flamingo: visual language model for few-shot learning, NeurlPS, 2022.
Gao, P., Zhang, R., Liu, C., Qiu, L., Huang. S., Lin, W., Zhao, S., Geng, S., Lin, Z., Jin, P., et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. ICML, 2024.
Arnab, A., Dehghani, M., Heigold, G., Sun, C., Lučić, M., and Schmid, C. Vivit: A video vision transformer. In ICCV, 2021.
Goyal, R., Kahou, S. E., Michalski, V., Materzyúiska, J., Westphal, S., Kim, H., Haenel, V., Fruend, I., Yianilos, P., Mueller-Freitag, M., Hoppe, F., Thurau, C., Bax, I., and Memisevic, R. The "something something" video database for learning and evaluating visual common sense. arXiv, 2017.
Bachmann, R., Mizrahi, D., Atanov, A., and Zamir, A. Mul- timae: Modai multi-task masked autoencoders. In ECCV, 2022.
Bain, M., Nagrani, A., Varol, G., and Zisserman, A. Frozen in time: A joint video and image encoder for end-to-end retrieval. In ICCV, 2021.
Han, J., Zhang, R., Shao, W., Gao, P., Xu, P., Xiao, H., Zhang, K., Liu, C., Wen, S., Guo, Z., et al. Imagebind- Ilm: Multi-modality instruction tuning. arXiv, 2023.
Bardes, A., Garrido, Q. Ponce, J., Rabbat, M., LeCun, Y., Assran, M., and Ballas, N. Revisiting feature prediction for learning visual representations from video. TMLR, 2024.
Han, J., Gong, K., Zhang, Y., Wang, J., Zhang, K.,, Lin, D., Qiao, Y., Gao, P., and Yue, X. Onellm: One framework to align all modalities with language. In CVPR, 2024.
Cha. J., Kang, W., Mun, J., and Roh, B. Honeybee: Locality- enhanced projector for multimodal 1lm. In CVPR, 2024.
Huang, B., Wang, X., Chen, H., Song, Z., and Zhu, W. Vtimellm: Empower Ilm to grasp video moments. In CVPR, 2024.
Chen, F., Han, M., Zhao, H., Zhang, Q. Shi, J., Xu, S., and Xu, B. X-IIm: Bootstrapping advanced large language models by treating multi-modalities as foreign languages. arXiv,, 2023.
Jain, J., Yang, J., and Shi, H. VCoder: Versatile Vision Encoders for Multimodal Large Language Models. In CVPR, 2024.
Chiang, W.-L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang. Y., Gonzalez, E., Stoica, I., and Xing, chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL https:/ /ite://mmmms /mmsysm.s.////////////ss/////ll/ wrg/bloggl 2023-03-2 1113330-v22 30-vicuna/.
Jang, Y., Song, Y., Yu, Y., Kim, Y., and Kim, G. Tgif- qa: Toward spatio-temporal reasoning in visual question answering. In CVPR, 2017.
Vicuna: An open-source
Jin, P., Takanobu, R., Zhang, C., Cao, X., and Yuan, L. Chat-univi: Unified visual representation empowers large language models with image and video understanding. In CVPR, 2024.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W., Sutton, C., Gehrmann, S., et al. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 2023.
Kar, O. F., Tonioni, A., Poklukar, P., Kulshrestha, A., Za- mir, A., and Tombari, F. Brave: Broadening the visual encoding of vision-language models. In ECCV, 2024.
Chung. H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y., Fe- dus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., et al. Scaling instruction finettnnd language models. Journal of Machine Learning Research, 25(70):1-53, 2024.
Karamcheti, S., Nair, S., Balakrishna, A., Liang, P., Kollar, T., and Sadigh, D. Prismatic vlms: Investigating the design space of visually-conditioned language models. In ICMI., 2024.
10

---

Unifying Specialized Visual Encoders for Vidco Language Models
Lee, S.-H., Wang, J., Zhang, Z., Fan, D., and Li, X. Video token merging for long video understanding. Advances in Neural Information Processing Systems, 37:13851- 1 2024.
Liu, S., Fan, L.,, Johns, E., Yu, Z., Xiao, C., and Anandku- mar, A. Prismer: A vision-language model with multi- task experts. TMLR, 2024b.
Luo, R., Zhao, Z., Yang, M., Dong, J., Qiu, M., Lu, P., Wang, T., and Wei, Z. Valley: Video assistant with large language model enhanced ability. arXiv, 2023.
Lei, J. Yu, L., Bansal, M., and Berg, L. Tvqa: Localized, compositional video question answering. In EMNLP, 2018.
Lyu, C., Wu, M., Wang, I.., Huang, X., Liu, B., Du, 7., Shi, S., and Tu, Z. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv, 2023.
Lei, J., Yu, L.., Berg. T. L., and Bansal, M. What is more likely to happen next? iido-and--laguuae future event prediction. In EMNLP, 2020.
Li, B., Zhang, Y., Chen, L.., Wang, J., Yang, J., and Liu, Z. Otter: A mudti-modal model with in-context instruction tuning, arXiv, 2023a.
Maaz, M., Rasheed, H., Khan, S., and Khan, F. S. Video- chatgpt: Towards detailed video understanding via large vision and language models. In ACL, 2024.
Li, J., Li, D., Xiong, C., and Hoi, S. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In JCML, 2022.
McKinzie, B., Gan, Z.,, Fauconnier, J.-P., Dodge, S., Zhang, B., Dufter, P., Shah, D., Du, X., Peng, F,, Weers, F., Belyi, A., Zhang, H., Singh, K., Kang, D., Jain, A., Hè, H., Schwarzer, M., Gunter, T., Kong, X., Zhang, A., Wang, J., Wang, C., Du, N., Lei, T., Wiseman, S., Yin, G., Lee, M., Wang, Z., Pang, R., Grasch, P., Toshev, A., and Yang. Y. Mml: Methods, analysis & insights from multimodal Ilm pre-training. ECCV, 2024.
Li, Li, D., Savarese, S., and Hoi, S. B Bootstrapping language-ing image pre-training with frozen image encoders and large language models. In ICML, 2023b.
Li, K., He. Y., Wang. Y., Li, Y.. Wang, W., Luo, P., Wang, Y., Wang, L.., and Qiao, Y. Videochat: Chat-centric video understanding. arXiv, 2023c.
OpenAI, R. Gpt-4 technical report. arxiv 2303.08774, View in Article, 2(5), 2023.
Li, K., Wang, Y., He, Y., Li, Y., Wang, Y., Liu, Y., Wang, Z., Xu, J., Chen, G., Luo, P., et al. Mvbench: A compre- hensive multi-modal video understanding benchmark. In CVPR, 2024a.
Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec. M., Khalidov, V., Fernandez, P., Haziza, D., Massa, F., El-Nouby, A., Howes, R., Huang, P--., Xu, H., Sharma, V., Li, S.-W., Galuba, W., Rabbat, M., Assran, M., Ballas, N., Synnaeve, G., Misra, I., Jegou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. Dinov2: Learning robust visual features without supervision. In TMLR, 2023.
Li, Y., Wang, C., and Jia, J. Llama-vid: An image is worth 2 tokens in large language models. In ECCV, 2024b.
Lin, B., Zhu, B., Ye, Y., Ning, M., Jin, P., and Yuan, L. Video-Ilava: Learning united visual representation by alignment before projection. In EMNLP. 2024.
Panagopoulou, A., Xue, L., Yu. N., Li, Li, D., Joty, S.. Xu, R., Savarese, S., Xiong, C., and Niebles, J. X-instructblip: A framework for aligning x-modal instruction-aware representations to Ilms and emergent cross-modal reasoning. arXiv, 2023.
Lin, Z., Liu, C., Zhang, R., Gao, P., Qiu, L., Xiao, H., Qiu, H., Lin, C., Shao, W., Chen, K., et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv, 2023.
Paträucean, V., Smaira, I., Gupta, A., Continente, A. R., Markeeva, L., Banarse, D., Koppula, S., Heyward, J., Malinowski, M., Yang, Y., Doersch, C., Matejovicova, T., Sulsky, Y., Miech, A., Frechette, A., Klimczak, H.. Koster, R., Zhang, J., Winkler, S., Aytar, Y., Osindero, S., Damen, D., Zisserman, A., and Carreira, J. Perception test: diagnostic benchmark for multimodal video models. In NeurlPS, 2023. URL https: opwnreview.eeeeeee/eeeeee n forum 21-HHEGXPPPPPPPG
Liu, H., Li, C., Wu, Q. and Lee, J. Visual instruction tuning. In NeurlPS, 2023.
Liu, H., Li, C., Li, Y., Li, B., Zhang, Y., Shen, S., Llava-next: Improved reason- ing, ocr, and world knowledge, January 2024a. URL https: /atp:://1lavavll..m wtt..vvw..gttu...bbbbb.boo 2024-00 11:4-0133-- 11-330-1aam 1lova next/.
and Lee, Y.
Liu, P. J., Saleh, M., Pot, E., Goodrich, B., Sepassi, R., Kaiser, L., and Shazeer, N. Generating wikipedia by summarizing long sequences. In ICLR, 2018
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable

---

Unitying Specialized Visual Encoders for Vidco Language Models
visual models from natural language supervision. In ICMI., 2021.
video foundation models via generative and discrimina- tive learning. arXiv, 2022.
Ryali, C., Hu, Y.TT, Bolya, D., Wei, C., Fan, H., Huang, P.-Y,, Aggarwal, V., Chowdhury, A., Poursaeed, O., Hoff- man, J., Malik, J., Li, Y., and Feichtenhofer, C. Hiera: A hierarchical vision transformer without the bells-and- whistles. In ICML, volume 202 of ICML 23, pp. 29441- 29454, Honolulu, Hawaii, USA, July 2023. JMLR.org.
Wang. Z., Wang, L., Zhao, Z., Wu, M. Lyu, C.,, Li, H., Cai, D., Zhou, L., Shi, S., and Tu, Z. Gpt4video: A unified multimodal large language model for Instruction- followed understanding and safety-aware generation. In CoRR, 2023.
Xiao, J., Shang, X., Yao, A., and Chua, T.-S. Next-qa: Next phase of auestion-answering explaining temporal actions. In CVPR, 2021.
Song, E., Chai, W., Wang, G., Zhang, Y., Zhou, H., Wu, F., Guo, X., Ye, T., Lu, Y., Hwang, J.-N., et al. Moviechat: From dense token sparse memory for long video un- derstanding. In CVPR, 2024.
Xu, D., Zhao, Z., Xiao, J., Wu, F,, Zhang, H., He, X., and Zhuang. Y. Video question answering via gradually refined attention over appearance and motion. In ACM Multimedia, 2017.
Su, Y. Lan, T., Li, H., Xu, J., Wang, Y, and Cai, D. Pandagpt: One model to instrution-follon them all. arXiv, 2023.
Xue, H., Sun, Y., Liu, B., Fu, J., Song, R., Li, H., and Luo, J. Clip-vip: Adapting pre-trained image-text model to video-language representation alignment. In ICLR, 2023.
Sun, G., Yu, W., Tang, C., Chen, X., Tan, T., Li, W., Lu, L.., Ma, Z., and Zhang, C. Fine-grained audio-visual joint representations for multimodal large language models. arXiv, 2023a.
Yu, E., Zhao, L., Wei, Y., Yang, J., Wu, D., Kong, L., Wei, H., Wang, T., Ge, Z., Zhang, X., et al. Merlin: Empowering multimodal Ilms with foresight minds. In ECCV, 2024a.
Sun, Q. Fang, Y., Wu, L., Wang, X., and Cao, Y. Eva- clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303
2023b.
Yu, S., Cho, J., Yadav, P., and Bansal, M. Self-chained image-language model for video localization and question answering. NeurlPS, 2024b.
Tong, S., Brown, E., Wu, P., Woo, S., Middepogu, M., Akula, S. C., Yang, J., Yang, S., Iyer, A., Pan, X., al. Cambrian-1: 1: A fully open, vision-centric exploration of multimodal Ilms. arXiv, 2024a.
Yu, Z., Xu, D.,. Yu, J., Yu, T., Zhao, Z., Zhuang, Y., and Tao, D. Activitynet-qa: A dataset for understanding complex web videos via question answering. In AAAI, 2019.
Tong, S., Liu, Z., Zhai, Y., Ma, Y., LeCun, Y., and Xie, S. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In CVPR, 2024b.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. In ICCV, 2023.
Touvron, H., Lavril, T., Izacard, G.. Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, F., Azhar, F., Rodriguez, A., Joulin, A., Grave, E., and Lam- ple, G. LLaMA: Open and Efficient Foundation Language Models, February 2023a.
Zhang, H., Li, X., and Bing, L. Video-llama: An instruction- tuned audio-visual language model for video understand- ing. In EMNLP, 2023.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv, 2023b.
Zhang, R., Gui, L., Sun, 7., Feng, Y., Xu, K., Zhang, Y., Fu, D., Li, C., Hauptmann, A., Bisk, Y., and Yang, Y. Direct preference optimization of video large multimodal models from language model reward, 2024a.
Wang, H., Ye, Y., Wang, Y., Nie, Y., and Huang, C. Elysium: Exploring object-level perception in videos via milm. In ECCV, 2024a.
Zhang, R., Han, J., Zhou, A., Hu, X., Yan. S., Lu, Li. H., Gao, P., and Qiao, Y. Llama-adapter: Efficient fine- tuning of language models with zero-init attention. In ICIR, 2024b.
Wang, S., Zhao, Q., Do. M. Q., Agarwal, N., Lee, K., and Sun, C. Vamos: Versatile action models for video under- standing. In ECCV, 2024b.
Zhao, L., Gundavarapu, N. B., Yuan, L., Zhou, H., Yan, S., Sun, J. J., Friedman, L.,, Qian, R., Weyand, T., Zhao, Y., et al. Videoprism: A foundational visual encoder for video understanding. ICMI., 2024.
Wang, Y., Li, K., Li, Y. He, Y., Huang. B., Zhao, Z., Zhang, H., Xu, J., Liu, Y., Wang, Z.,, et al. Internvideo: General
12

---

Unifying Specialized Visual Encoders for Vidco Language Models
Zhao, Y., Gu, A., Varma, R., Luo, L., Huang, C.-C., Xu, M., Wright, I.., Shojanazeri, H., Ott, M., Shleifer, S., Desmaison, A., Balioglu, C., Damania, P., Nguyen, B., Chauban, G., Hao, Y., Mathews, A., and Li, S. Pytorch fsdp: Experiences on scaling fully sharded data parallel, 2023.
Zhu, B., Lin, B., Ning, M., Yan, Y., Cui, J., Wang, H., Pang, Y., Jiang, W. Zhang, J., Li, Z., et al. Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. In ICLR,,, 2024a.
Zhu, M., Zhu, Y., Liu, X., Liu, N., Xu, Z., Shen, C., Peng, Y., Ou, Z., Feng, F., and Tang, J. A comprehensive overhaul of multimodal assistant with small language models. In AAAI, 2024b.
13

---

Unitying Specialized Visual Encoders for Vidco Language Models
A. Appendix / Supplemental Material
A.I. Limitations
A.2. Model Details
A.2.1 Baseline Encoder and LLM Details
A.2.2 MERV Encoder Details
A.3. Detailed Experimental Results
A.4. Something-Something v2 Details
A.4.1 Something-Somethingg OpenEnded
A.4.2 Something-Something v2 Temporal
A.5. Additional Experiments and Analyses
A.5.1 Adding More Encoders
A.5.2 Attention Weights
A.6. More Qualitative Results
A.1. Limitations
Our work is based on the LLM, LLaMA-2 7B model (Touvron et al., 2023b), and as with many other VideoLLM models, the performance of our method hugely dependent on the capabilities of the L.M model, and better-performing models often demand significantly more computation. MERV requires an LLM and running multiple encoders, which can be computationally intensive and can lead out-of-memory errors in lsourece-lmmied settings (while our efficient implementation alleviates the issue). While FSDP (Zhao et al., 2023) allows us easily and effectively train larger models across multiple GPUs than would otherwise be possible, its generality also makes it difficult for us to design tailored sharding strategies that would maximize the performance of our model. However, with future improvements to data parallelism, our model can still benefit greatly and run even more efficiently. Also, despite the improved speeds, there is still an upper bound for what constitutes a reasonable training time that still allows us to test many of our design assumptions, which limits the scale and number of experiments we can run. While we show that our method can successfully leverage information from different visual encoders, nevertheless if the encoders themselves are limited in video understanding capability, MERV cannot fully compensate for that.
A.2. Model Details
1. BASELINE ENCODER AND LLM DETAILS
Table 3, Visual Encoder and LLM Information.
Model
Visual Enceder
LLM
(contributor, LLaMA-1 78 (Tonvion al., 20234)
AdaMAAAAaaer (Zhang at. 2024b)
CLIP (Radfard et al., 2023)
Vide--I1aA (Zhung 2023)
VIT-G (Sun et 2023b) BI-P-2 Q-Remer (1i 20236)
Viauna-BB v0 (Chiang al, 2023)
Video-ChanGPP (Mazz
(Radford al. 2121)
Vicuna-BB (Chiang
SeNiLA (No 2024b)
(EVA-CEV (Son al., 20236) BLJP--2 Q-Foemer (1i et al., 20236)
RanT5-XT (3B) (Ching et 2024)
LLaMA VID 70 (Li al., 2024b)
EVA (Fang al., 2023)
Vicuna: 7B v1.5 (Chaang et al., 2023)
LLaMA-VID-13B
(Fang
Vicuna-138 (Chiang
2023)
Video-L1aVA* (I al., 20044)
LangugeBBind etal. 2024g)
Vicuna-BB (Chiang et si., 2023)
A.2.2. MERY ENCODER DETAILS
Here, we detail the visual encoder details, LLM, and the training objectives. We plan to release our full code including training and evaluation as well as all model checkpoints for the camera-ready version of the paper.
LanguageBind We use the code from the original author, using the pre-trained kanguaggesind/tanguggess Video me rge uploaded on huggi faggeee..
weight
14

---

Unifying Specialized Visual Encoders for Vidco Language Models
Table Encoder Information. Detailed information about the four encoders used in our experiments. They represent broad coverage of visual information and training objectives.
Model
Architecture
Expertise
Training Dutasets
Training Objective
I LanguageBiid (Zhu et al., 2024a)
VIT-1/14
Viden+I anguage
10M. five-modal video examples
Contrastive
DINOv2 (Oquab al., 2023)
ViT-I/14
Spatial
LVD-142M
Self Supervised Supervised
VIVIT (Arnab al., 2021)
VIVIT-B/IGX2
Actions/Temporal
Kineties-4006600, short videos 4B curated image/text pairs
IPgL.P (Zhai et al., 2023)
WTT-/116
Image+I.anguage
Contrastive
DINOv2 As DINOv2 is an image-model, we get embedding per frame, and concatenate them to be a video embedding. We use Vi Ttarge model, pre-trained on LVD-142M dataset, and take the penultimate layer for the embeddings. Specifically, .idvl4144 ivd1 42m
we use t timm's 's
ViViT We use Visbase as our backbone, pre-trained on Kinetics-400 dataset. Specifically, we use google/vivit- 1ttp-o/xxx-iiieiiie000000 uploaded on huggingface. We use featurizer output as the video embed- ding.
SigLIP As SigLIP is an image-model, we get embedding per frame, and concatenate them to be video embedding. We use ViTusse as our backbone, and take the penultimate layer for the embeddings. Specifically, we use timm's vit base patch16 siglip 224
We also considered multiple other options for encoders, such as CLIP-VIL ViP (Xue et al., 2023) for our rideo-language contrastive expert, V-JEPA (Bardes et al., 2024) and Hiera (Ryali et al., 2023) for our pure video model, and CLIP (Radford al., 2021) for our image-language contrastive expert, but found that our choices performed better overall.
A.3. Detailed Experimental Results
Here we tabulate the full experimental results that was abbreviated from the main paper. The first table (Table 5) ablates the different training recipes we tried for MERV, with extended discussion in Section 4.2.3.
Table 5. Ablation of training stage recipes. We explore different training recipe strategies, starting with the standard LLaVA recipe which Video-LLaVa VA adopted, along with some other variations.
MSVD-QA
MSRYTT-QA
TGIF-QA
Perception Test
AetivityNet-QA
Methods
Ace
Score
Soore
Score
Acc
Acc
Score
MERV (frozen)
70.97
3.76
59.03
3,25
51.1
3.26
46.21
50.87
3.34
MERV. Vidco-LLaVA recipe
70.92
3.78
58.74
3.25
51.67
3.27
47.48
50.42
3.33
MFRV (full)
70.48
3.79
57.25
3.24
$1.39
3.28
48.41
49.93
3.33
MERV. mixed Stage
55.14
3.08
51.53
45.65
Pre-fusion projector details. The image-level projectors are similar to those described in MM-1 (McKinzie et al., 2024): 2D adaptive average pooling, shallow attention resampler similar to Perceiver Resampler (Alayrac et al., 2022), and convolutional pooling with 3 RegNet blocks on both sides of an average pool layer such as the C-Abstractor in Honeybee (Cha et al., 2024). For video-level projectors, we use a 3D average pool, where we pool to the same spatial dimension but furthermore pool the frame dimension by 2, and 3D convolution where we add a single 2 x x 3 convolution before the same average pooling. For all projectors, we project to the same number of tokens x h x w, using an adaptive average pool h x latent tokens for the attention resampler.
We also provide the full metric numbers in Table 6a, 6b and 6c for our ablations that are described in Sections 4.2.1 and 4.2.2.
Note on TGIF-QA Video-ChatGPT's and Video-LLaVA's author-reported numbers on TGIF are incomparable as they were on a subset of the dataset. See https: / //////gii com/ htt.tttt..cco/pp....aaooooooooe. LLaVA/ Lssues/37 37.
15

---

Unitying Specialized Visual Encoders for Video Language Models
Table Full design choice ablation numbers. Detailed experimental results Tables 2a, 2b, We highlight our defaults orange and bold the best results.
Pre-fusion output token. We ablate the (a) Pre-fusion projectors. * 16 frames instead of 8, Top two rows are projector- optimal token per frame the pre-fusion free baselines.
projector.
Projector
MSVD
MSRVTT
TGIF
Perc.
Params
FLOPS
Tkns
MSVD
MSRVTT
TGIF
Perc.
257 tok
68.47
55.81
48.62
46.14
61.94
54.64
41.41
42.85
class tok
65.98
55
43.7
43.51
64.47
55.72
45.32
43.31
2D Avg
68.23
56,92
48.99
45.69
2.1M
16
67.23
56.44
47.75
43.18
64
69.08
58.00
50.01
46.34
2D Avg*
69.08
50.01
46.34
4.2M
100
68.38
57.47
48.78
45.56
2D Attn
65.76
55,23
43.35
44.14
12.7M
9.7G
144
57.73
48.81
43.94
2D Conv
67.48
56.78
47.6
45.04
237M
241G
256
68.46
57.72
48.66
43.51
3D Avg*
68.62
57.2
49.59
44.95
4.2M
3D Conv
68.56
57.03
49.28
46.81
113M
232G
(c) Feature fusion strategy. We compare our feature fusson strategy with concatenating the visual embeddings either token sequence dimension or the channel dimension, learning an optimal emhodding mixture weights, and training with equal 25% mixture of visual embeddings.
Strategy
MSVD
MSRVTT
TGIF
Pere.
FLOPs
Cross-Attn
70.97
59.03
51.1
46.21
17.19 T
Concat (Seq.)
66.99
56.95
48.20
45.67
43.09
Concat (Ch.)
70.02
58.08
51.1
47.36
16.29
Learnable W
68.06
56.54
48.82
46.6
16.24 T
25% Mixed
68.38
56.99
47.71
43.66
16.39 T
Table 7. Effeet Encoder. Detailed results Figure
MSVD-QA
MSRVTT-QA
TGIF-QA Acc
Perception Ace
ActivityNet-QA
Avg
Methods
Acc
Score
Acc
Score
Score
Acc
Score
Acc
All 4 encoders
70.97
3.76
59.03
3.25
51.10
3.26
46.21
50.87
3.34
55.64
w/o LanguageBind
68.52
3.69
57.10
3.19
50.20
3.23
45.23
49.78
3.31
54.17
w/o DINOv2
69.75
3.74
57.70
3.23
49.94
3.23
46.57
51.43
3.34
55.08
w/o VIVIT
70.12
3.75
58.26
3.23
50.45
3.22
46.94
51.36
3,33
55.43
w/o SigLIP
69.85
3.74
57.55
3.22
50.27
3.22
46.20
50.06
3.32
54.79
Table MERV Captures Single Encoder Performances. Detailed experimental results of Figure 4a.
MSVD-QA
MSRVTT-QA
TGIF-QA Acc
Perception Acc
ActivityNet-QA
Avg Acc
Params Overall
Methods
Acc
Score
Acc
Score
Score
Acc
Score
FLOPs
MERV
70.97
3.76
59.03
3,25
51.10
3.26
46.21
50.87
3.34
55.64
17.19 T
7686.0 M
LangBind
68.47
3.71
55,81
3.16
48.62
3.19
46.14
44.72
3.17
52.75
41.3 T
7147.0 M
DINOv2
65.44
3.62
53.46
3.09
41.53
2.96
42.73
43.39
3.09
49.31
40.88 T
7046.0 M
ViViT
59.95
3.43
51.81
3.05
38.1
2.84
40.2
43.98
3.16
46.81
27.12 T
6830.0 M
SigLIP
66.68
3.64
56.41
3.16
48.22
3.16
45.09
49.41
3.31
53.16
31.08 T
6834.0 M
16

---

Unifying Specialized Visual Encoders for Vidco Language Models
Table 9. Performance on WH-words. Detailed experimental results Figure 6.
MSRVTT what
MSRVTT-bow MhoV
MSRVTT when MSVD-whal
MSVD-whi
TGIF-whut
TGIF-how
TGIF-
MERV
50.62
77.17
83.96
72.23
62.68
84,62
49.44
53.33
65.34
VIVIT
43.06
70.43
78.90
68.54
50.10
75.90
32,90
50.10
50.62
DINOv2
44.54
73.00
76.95
65.73
55.71
82.12
37.14
50.69
58.40
LanguageBind
46.89
72.53
46.23
53.24
Siyl.IP
47.96
74.41
84,21
71,20
57.17
82.23
45.65
53.25
60.21
A.4. Something-Sometting v2 Details
INSTRUCTION
How & the cbject in the vides being stcracted with?
ISTRUCTION:
the the internteed
with?
NER
object sione betig interacted wen parsors shand hand
VERV
onen the object, peectaaae mall siaces E the
penuents hand, and on
surface. hand thes meves dbea anound surfacs possibly
ba
then
Video-LLaWk:
the ustnng match interact with They
aeen RBM ng match over stgect than usie
talls They anso parne Eeggenate
person the videe ln uning hiliihh paint cank palleh FONTS
LLaMAAR::
The object the vidso baing inter acted wit by person who seen placing
(URSS Bestte Besttiaa apsly
shoun apai..
the obrect plaste bottle interacteed
with?
MERV
peraon The person seen ba bottle and parnting it while
object
pemoo's hand The penson is seen tilliing the peniet in thei fand and mov
Vidco LLAVA
The the video inter interaccing with the objjctt by Sounting woberiinuit
Video-LLash
hard pand untng P ece paper: They also aenc
the
IL
Bieer BUANE perion lhen BIERK UR Sles and
vidio pealing bottle
towards the camera. They than use the pencil to jeal on prgies aupps
INSRUCTTIN ION:
the ebrect the interaceed wht??
MERY:
samatn strming thair
interacting
sowi
parsor's hand. The pesson litasiing butions on shanng which Blaced
seon Becosing someetenn out ll bow and putling & into thei mouth.
Vidco-LL1A:
pusson in the vidco soen hord the pressing bustonsloo Besnup this alfani
DoMi
jiling
Bultons on R.
thdir
dMect
with?
WET Barn goneen The peeson seen hobding adowrrs and cur the
ozch and turning the pages. sotson s alsa sean do monning soud
vidio
Mideo-11
sho seen undin knitee Secc
vidio
V-dco-1L34A
The porson in the vidco soen Tioe through the of the book Uani
Stoowee Cur Bagad The person in allo oeen us the' hande iieeett
the 53305 book They are ase see lotato an camate andl H803.
the
chde
shrough
Figure 8. Samples of MERV in SSv2. Due to our design, our method shows better temporal action understanding than other VideoL (Top two rows) However. the difficulty the task. we failure cases for VideoLLMs. (Bottom two rows)
17

---

Unitying Specialized Visual Encoders for Vidco Language Models
Pushing [something] so that it slightly moves
Pashing (something) from right to left
Pushing [somncthing] from left to right
Moving (something] down
Moving (something] up
Stutting (semething] into somchhing]
Taking [onc of many similar things the table]
Covering (somethingg with (omeching]
Showing that sommithing] ia empty
Pluesing [something into [something]
Figure Example video of Homeehhin-SSomethngg V2. that ViVIT show better performance in classes where temporal movement is critical for solving the task (Top row), while SigLIP performs better when the action can be inferred from the image without temporal information (Bottom row).
A.4.1. SAMETHING-SOMETHINGG V2 OPENENDED
Table 10. Performance on Srnnething-Something V2 OpenEnded. These are the performance in shown Figure 4b
MERV
MERV-Full
LanguageBind
DinoV2
VIVT
SigL.P
LLaMA Vid- Vid-MA-Vdd-7
LLaMA Vid-13B
aad
Smth-Smtth O-mmpollll
6.82
3.63
3.88
5.50
5.57
Smth-Smth V2-0E
17.70
20.65
13.83
11.03
10.53
13.84
16.47
15.62
19.18
Suth-Smth V2-MCO-
36.84
40.65
30.58
28.08
39.77
25.39
27.14
17.89
22.47
Smuth Smuth V2 MCQ
42.01
39.76
36,82
33.06
26.78
34.86
36.63
39.43
23.14
Additionally, we evaluate Something-Something V2 as an open ended QA task, where the question is "How is the object in the video being interacted with?", and the answer is expected to similar to the class label. We use Video-ChatGPT (Maaz al., 2024)'s LLM evaluation for validating the VideoLLMs' output. Table 10 tabulates the results. We see similar conclusions as with MCQ where ViViT excels at temporal subset than other single-encoder LLMs, but failing to match the performance in full dataset. Nonetheless, method shows the best performance overall.
A.4.2. SOMETHING- COMETHING-SOMETTING SOMETHING v2 TEMPORAL
The 12 selected classes are as following:
Approaching (something] with your camera
Turning the camera downwards while filming [something]
Turning the camera left while filming [something]
Turning the camera right while filming [something]
Turning the camera upwards while filming [something]
Moving away from [something] with your camera
Moving [something] away from the camera
Moving [something] towards the camera
Pulling [something] from left to right
Pulling [something] from right to left
Pushing [something] from left to right
Pushing [something] from right left
18

---

Unifying Specialized Visual Encoders for Video Language Models
INTRUCTION
Describe object
ISTRUCTTION:
Deneribe
INSTRCTION TON:
Descrbbe the dbject
The noject yeliow ball with sitting posden dedd
Frower top The box made carobood
The abject Email box with 20l
MERV
The db pet imag presn papart Whith paccer ATMO bex posanll for
VINTT
The onject ii Emall toy with a and whihe there a small bal noorts
Fower Incale
VWTT:
The object &s Brown Bus with white led
WAT
The de oot aamaal seuare shipll piace passel Bis Bel boing
SKGL
The object in agge orange ball that sitting
halld
ISB: arhan i ettiing on carpet
SIGLIP:
he
The gresn
parsen
that beind
Woiing in atject STrDK sutfere undil
INSTR.CTIIN::
Hiww is the object in the vilno bring ierrected with?
INSTRUCTION:
abject the video being beractee with?
Opening
Tearing an déject into two pieces
vd ow tall placing ther hand and thoy rolling
The 3nd bd slcced on co mpputer montter.
vitee,
seen interacting with
(eed Bieeiing the bos and takinn out a pink item and put bock down.
video,
inmeectiing
MERK
The 48 in the vidoo boing inturacted th
papes The
diiiing and toiing
hilkking Drc
in hand and migg to out paperr. whichh than
VWTT:
The ooject in the video small toy car that oeen puahting the car aroud on table, and the
VITT
The semenn the videa is interacinng g seit the indiee They then proceed unseas the ttem
WXTT:
The san video Ball di ect, eriic piece papee.
sciasora
SIGLIP:
The cá aet the being interacted who holding and araapinng green paper The perven then tapas papser
recoon who i seen helsinn it and mieiin bacc 2nd and Thros tioo gool
person who is seen holding if and slacino en carpet Rowmepphng papst and tape stur.
Wmap
Figure 10. Example LL output on Something- SoweetiingSSmttinggggg v2. While SigLIP performs better on object and scene recognition, it fails to understand temporal actions. VIVIT fails on the details of object recognition, but has better understanding in temporal movements.
A.5. Additional Experiments and Analyses
A.5.1. ADDING MORE ENCODERS
Additionally, test with more encoder Hiera (Ryali et al., 2023). The checkpoint was pretrained using the Masked Autoencoder (MAB) self-supervised learning technique Kinetics 400 (K400), and then finetuned on K400 using supervised learning.
Table 11. Performance with Hiera (Ryali et al., 2023).
MSVD-QA
MSRVTT-QA
TGIF-QA
Perception
ActivityNet-QA
Methods
Acc
Score
Acc
Score
Acc
Score
Acc
Acc
Score
VideoLLaVA
67.74
3.69
56.90
3.18
47.99
3.17
47.08
3.27
44.22
MERV
70.97
3.76
59.03
3.25
51.1
3.26
50.87
3,34
46.21
MERV, VIVIT replaced with Hiera
69.68
3.74
57.64
3.22
50.38
3.24
50.24
3.34
47.50
MERV Hiera
69.67
3.72
58.26
3.23
50.32
3.22
51.23
3.34
46.23
ViViT Single Encoder LLM
59.95
3.43
51.81
3.05
38.10
2.84
43.98
3.16
40.20
Hiera Single Encoder LLM
55.38
3.28
49.21
2.95
36.02
2.76
44.01
3.15
40.20
The results are presented in the Table 11, with the best result in bold. Replacing VIVIT with Hiera demonstrates improvements on the fine-grained spatio-temporal reasoning benchmark, Perception Test, with accuracy gain of 1.29%. Similarly, adding Hiera yields improvement on ActivityNet, achieving 0.36% increase in accuracy. However, other benchmarks, the original MERV remains the strongest model. Overall, observe significant performance improvement when training with Hiera, which aligns with expectations, since Hiera is under the same paradigm as ViVIT, functioning as temporal expert trained on short videos. We also hypothesize that Hiera is more sensitive to the temporal stride than ViViT, as ViViT can reasonably deduce motion from uniformly sampled frames. We expect performance to improve if we incorporate encoders trained on different paradigms and data sources or process a much greater number of frames simultancously, which we will leave for future work.
A.5.2. ATTENTION WEIGHTS
Our analysis in Section 5.1 looks at how different cross-attention weights activate most different videos, illustrated in Figure 11.
A.6. More Qualitative Results
We present additional qualitative results in Figure 12 and Figure 13.
19

---

Unifying Specialized Visual Encoders for Vidco Language Models
LanguageBind
DINOv2
VIVIT
SigLIP
iskal Cliiskal Cliiskal Cli iskal Cli
THE FOOD STORE
Caplyyal 02
OOD STORE
Bio BETES
Figure 11. Videos that give the highest attention weight for each of the encoders. The right-most column shows the average frame of the video.
20

---

Unifying Specialized Visual Encoders for Vidco Language Models
Diserioe
vides.
porforming
wearing
standing ng ina lising room and ser nging the ramrte atound wille ng.
and real hear shapel balloon, also Mack siin black
RO
What game playin?
background?
HEER
thene
INSRUCTTON ION:
What room
INSTRIACTION
What country do you think the senson parformng?
MERM
The man is in lvingg mom.
MEWW
Boced on the mape, theperson & per ferfmiing France, they are wearing lower Bfidl famous unddmans For france, and populor toutse disti sltnn The S48 ng.
RUITT IoNe
playing alcne?
supgest
people
bkay.
Describ person deing video
ISTRUCTTIN
Descrize what person doing videa.
Now ppor
MEAV:
person viceo is pert orming lop atrecch by litinn their right ding
stand the mat and talking cameia while showing how olencSc.
the
INSTRCT SOTRUCTION:
Bescribe the tiches that thes perton sver
person wewttng
and baric parts
HERW
ofe
HERV
Based
seen pertorming oen routie, speeti Iy on patted Hapr.
roam Surther
that gym ora wertout spare. man seering tue
indicate
perticpating
unnee to designed physiri antivity ecertio with man uning workour
INSTRUCTION
Ouso be what the person Jiing in video.
RER::
The posson in the viice parfo-ming ming worcow rouline on a ba sinnce board, They ma ntaining thai bulurie.
INSTRUCTION
What the cofor the eh the person awaring in the
ng
INSRUCTIOO
Descriie
videe.
INSTRUCCTIO
color the salance board that
MERV
tee amms lege while danding.
IERR
The ca ofe sa ante bound that the person is bakk
Describ room person mude
gam room
prncrt
and gom mats.
Figure 12. Samples of MERV in video understanding.
21

---

Unifying Specialized Visual Encoders for Vidco Language Models
ASTRUCTION
The déject vide beng
PS PSTTUCTION:
How is the oct the dioo being Roro ded with
with?
object uip tod person The
loombrush
person whs & Proning bil nis cue Ennce and cup Mnocked ond each tims
wus anng and parson Ioennrram Sainng used clean
MERV:
The ab videc plastic bottis that
ri interacted
corging the bostle pointing serson
pesibars, sucs
analing (i
depect nubbing perao table appears yet Pearact
rnpect
VITE:
Thi ausson UNE todoo is us tool pushi
Claan the lis sur loce the vadie. They and solet Birk 34a
ball
the
Bntlo the 20
5K2 P.
The ostect in the videa and bat that bein oeen pirkingg the sall and throding in back down
SISUP:
object in dids tube Brandilg and
SICLIP
cb video parson. glasn)
of water which
videe helsing
interacted Bhuning shng bett
than pesseed Realy (ip soneed barm eventiy
slyingg with
ano bette glacss
bare benitie The
languageline int
The pawas vidne Being escanis
Languagetins
they hand and using to bruch their teeth They
Languagellidd
The person wass is haleing gi the engectin
The osjeet in the utoee ssing Intervntedd with By elaceng on the tabii)
tnto the oheck and shake fl
They
bong Icead Bien Wi pol Jown with napkin)
DINOV2:
Thu pe widce s holding selectif
into and ditncinng sing fro
ASTRUCTION
what naibll
yullow
ISTRUCTION::
what man running through the weeds n813
MER:
The punson nn yellow bald.
runting through woods bow and crow.
RS INC TOW:
how many poodle are Diare video
The panas yelow weering Wotaae tambil
Thene only one parson
who
WiTT:
man a nuno through the woods arth
WITT:
There are Bat pros the die..
Langaageend
The posos Waye curting the grass with
9IS1
runn motomyyie
woods
SULIP::
There fro peodil in video.
DROY2:
The pasos Fow moving lown Ing Ins, & common requus LO02 mairfe unioom
Languagetins
Ieea 00
through
Languageliid:
two
DRNOv2:
The man is unm thruuh the woods with a
Figure 13. More qualitative results. MERV tends to show improved understanding in temporal hearyy videos as in Something- Something v2 dataset (Goyal et al., 2017) (Top Row), while retaining the performance on scenic understanding, seen from popular video bench- marks (Xu al., 2017; Yu et al., 2019) (Boltom Row).
22


Reviews:
[{"review_id": "FNNxWWvkbt", "reviewer": "Reviewer_ukCZ", "summary": "This paper tackles the problem of video-language understanding by introducing a multi-encoder strategy for constructing a comprehensive representation for videos. The authors claim that existing single-encoder methods can merely obtain a limited amount and type of visual information. Therefore, they proposed to leverage multiple encoders from different backbone families (assumed to have different and diverse capabilities) and map their representations into a unified space. Experimental results demonstrate the performance gains given by such multi-encoder design.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer ukCZ, we thank you for spending your time and effort on reviewing our work. We appreciate your positive comments on our practical design choices that can enhance the capabilities of existing Multimodal LLMs (MLLMs). Your concerns are with a) the **insights** gained from our paper, and b) the **novelty** of our method. We will address them as follows. \n\nWe posit that while our final method seems simple, it is the result of exploring >100 different configurations, not all of which worked well, which resulted in this simplicity. For example, while prior methods like VideoLLaVA (which our work primarily builds on) used full-resolution visual tokens, we found that combining 2D averaging with just 64 tokens/frame outperformed this base setting and was far more efficient (Table 2(a) and (b)). Note that in Table 2(a), only 2D/3D Avg and 3D Conv outperform this base setting, with 3D Conv doing so at a significant cost in parameters and FLOPs. Similarly, in Table 2(b), 64 frame tokens outperform the default 256 from prior works, with 144 being the only other better setting. Finally, also see our response to Reviewer 1K5Z discussing how even our choice of encoders is not simple. This shows that it is not a matter of simply adding encoders, as without caring for these details, the resulting method is easily worse and even less efficient than the original method. \n\nAs for novelty, our work is indeed the first to successfully leverage multi-encoders for video understanding MLLMs, as agreed by the Reviewer 1K5Z and 44Cd to be novel. We also believe there is merit in the detail and thought behind our paper. For example, we contribute a computationally efficient scheme for adding additional encoders which takes advantage of the distributed setups required today (Fig.3). We provide a very in-depth analysis of multi-encoder MLLMs, to a level of detail not taken by other works (Sec.3 and 4). We illustrate behaviors behind these large pretrained models reflecting their training data and inherent biases which are not commonly understood (Appendix, Fig.10) (e.g. SigLIP trained models are biased towards textual data). Additionally, one underrated choice we made was to deliberately focus on a single training mix so that we can control for data and understand these models and our choices. The video domain has many quirks not present in images, so discovering additional settings which work is nontrivial.\n\nFinally, we summarize our findings and share insights not covered in prior research:\n\nInsight 1: What works for integrating multiple RGB-encoders into a single VideoLLM:\n1. Select encoders from distinct backbone families.\n2. Align features spatio-temporally before fusion. \n3. 2D average pooling (no parameters) is the best pre-fusion projector.\n4. Optimal token size per frame is 64 (searched from 1 to 256).\n5. Stage 2-only tuning is fast (43% time) with similar results; unfreezing LLM in Stage 1 boosts alignment and benchmark scores.\n\nInsight 2: What doesn’t work:\n1. Assembling random encoders can hurt performance.\n2. No pre-fusion strategies lead to worse results and higher FLOPs.\n3. Other projectors (class token, 3D avg, perceiver-style attention, 2D/3D conv) perform worse and cost more.\n4. Too small/large token sizes per frame degrade performance.\n5. Alternative fusion methods (concat, equal weighting, learnable scalars) are suboptimal considering both performance and efficiency.\n6. Training only projectors and fusion in Stage 1 or mixing stages performs worse than explicit two-stage training.\n\nInsight 3: Computational time doesn’t scale linearly with respect to the number of visual encoders used, as one might initially think, thanks to multi-GPU model parallelism. Using more encoders adds very minimal overhead (Fig.3) since they can run in parallel and are small compared to the LLM.\n\nInsight 4: MERV outperforms prior works by up to 4.62% with minimal added parameters and faster training. It leverages multiple encoders effectively, while each encoder is truly contributing, without trading off performance between specializations as single encoder models do. This offers a free alternative to scaling training data and model understanding by scaling visual encoders instead.\n\nInsight 5: MERV provides better interpretability with the inner workings of visual encoders and the decision-making process of MLLMs. Its cross-attention weights activate on corresponding videos, allowing it to capture the visual strengths of different encoders and intuit both motion and general understanding simultaneously.\n\nSummary: As elaborated above, we are not simply making use of more encoders, nor did MERV just provide a simple strategy that most researchers would already know to be effective. As Reviewer 44Cd and 1K5Z agrees, MERV introduces novel direction and is the result of extensive effort and care in exploring the right design choices. We provided many insights into how to effectively encode information using a single/few encoder(s).", "claims_and_evidence": "The main claim in the paper is “existing single-encoder methods are capable of obtaining only a limited amount and type of visual information.”, which is somewhat reasonable, but it is challenging to proof it with any rigorous theoretical analysis.", "methods_and_evaluation_criteria": "Yes. From the application perspective, the proposed strategy can effectively enhance the capabilities of existing MLLMs.", "theoretical_claims": "The paper does not contain any proofs or theoretical claims.", "experimental_designs_or_analyses": "The experiments are conducted on public video understanding benchmarks, with well-designed metrics and fair comparisons. The experiment protocols are reasonable.", "relation_to_broader_scientific_literature": "The key contribution is to extend the existing Video-LLMs from single-encoder to multi-encoder, which I think is already a common strategy in enhancing visual perception capabilities in the entire CV community. This makes the novelty of the proposed scheme limited.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Generally, the paper provides some practical design choices and experiments on how to extend single-encoder models to multi-encoder styles. Since this is a simple strategy that most researchers would know of its effectiveness, I think the novelty of this paper is limited. It would be better to provide more insights about how to jointly encode more diverse information (using a single encoder or only a few smaller encoders) rather than simply making use of more encoders.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "The authors are encouraged to provide more strong justifications for the novelty of the proposed framework.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "pB7S16iUZJ", "reviewer": "Reviewer_1K5Z", "summary": "Traditional VideoLLMs typically rely on a single vision encoder, which restricts the model's ability to leverage the diverse strengths of different visual encoders. To overcome this limitation, this paper propose a novel framework that integrates multiple specialized vision encoders into a unified video representation model. The proposed approach employs encoders such as SigLIP, DinoV2, ViViT, and LanguageBind—each contributing unique capabilities in spatial, temporal, and multimodal understanding.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 1K5Z, we thank you for spending your time and effort on reviewing our work. We appreciate your recognition of the motivation, the novelty of our framework, extensive experiments, insights we provided to practitioners, and high reproducibility of our work. Your concerns seem to be with a) the **innovation** of our paper, particularly concerning our feature fusion and our relationship to ensemble methods, and b) our **reasoning behind our encoder selection** and the choice of picking more recent video encoders. Before we will address these in order, we want to point out an error in your review. **The supplementary material was provided in page 13~22**  which addresses some of your questions.\n\n**W1: Weak innovation about method, esp. around feature fusion and ensemble methods.**\n\nWe agree that ensembling is a fundamental concept in machine learning, and simply applying such methods without understanding how each unique domain affects the method in practice would not provide any new insights. While our method may seem simple at a first glance, there were many design decisions which were critical in enabling the strength of our results which are not apparent if one tries to simply scale prior works to multiple encoders; see our detailed response to ukCZ, where we discuss how our method provides new insights into practically building these ensemble methods. In contrast, prior works such as Prismer [1]—though not focused on RGB-based video understanding MLLMs—have failed to make a marked improvement over single-encoder baselines. \n\nAs for the feature fusion, our method is quite different from existing approaches, as those primarily fall under either channel concatenation or sequence concatenation, as mentioned in the paper text with references (Appendix A.2. Related Works). However, our method is still better or on par with these methods, while also providing extra, meaningful analyses into how the model is making these decisions as shown in Fig.10. By visualizing the videos that have the high attention weight of an encoder, e.g. videos with a lot of text have high attention weight on SigLIP, we gain insight of the data mix and training objective of each encoder. \n\n**Q1: Why these four encoders, and more recent video encoders?**\n\nWe chose these four encoders because we felt that they offered a broad coverage of both optimization objectives and training data, both of which are important for giving our model a wide breadth of coverage. Within each type of model, we performed experiments using a few different models, such as V-JEPA [2] for video, but ended up choosing ViViT based on final performance. We hypothesize that V-JEPAl is difficult to use for frozen adaptation, so we leave this as future work for others. The other models were chosen based on strong performance in other places. \n\nIn our Appendix A.6, we have conducted experiments using a newer SoTA video model Hiera-B+ [3], i.e. MViTv3, in both a four-encoder setting replacing ViViT and a five-encoder setting. \n\nFor your convenience, we list the results in the table below, with the best result in bold and the second-best in italics. Replacing ViViT with Hiera demonstrates improvements on the fine-grained spatiotemporal reasoning benchmark, PerceptionTest, with accuracy gain of 1.29%. Similarly, adding Hiera yields an improvement on ActivityNet, achieving a 0.36% increase in accuracy. However, on other benchmarks, the original MERV remains the strongest model. Overall, we observe no significant performance improvement when training with Hiera, which aligns with expectations, since Hiera is under the same paradigm as ViViT, functioning as a temporal expert trained on short videos. We also hypothesize that Hiera is more sensitive to the temporal stride than ViViT, as ViViT can reasonably deduce motion from uniformly sampled frames. We expect performance to improve if we incorporate encoders trained on different paradigms and data sources or process a much greater number of frames simultaneously, which we will leave for future work.\n\n| Method | MSVD -Acc | MSVD -Score | MSRVTT -Acc | MSRVTT -Score | TGIF -Acc | TGIF -Score | ActivityNet -Acc | ActivityNet -Score | Perception -Acc | \n | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | \n  | VideoLLaVA | 67.74 | 3.69 | 56.90 | 3.18 | 47.99 | 3.17 | 47.08 | 3.27 | 44.22 | \n | MERV | **70.97** | **3.76** | **59.03** | **3.25** | **51.1** | **3.26** |  *50.87* | **3.34** | 46.21 |\n| MERV,  ViViT replaced with  Hiera | *69.68* | *3.74* | 57.64 | 3.22 | *50.38* | *3.24* | 50.24 | **3.34** | **47.50** | \n | MERV + Hiera | 69.67 | 3.72 | *58.26* | *3.23* | 50.32 | 3.22 | **51.23** | **3.34** | *46.23* | \n\n[1] Liu et al., Prismer: A Vision-Language Model with Multi-Task Experts, 2023. \\\n[2] Bardes et al. \"Revisiting feature prediction for learning visual representations from video.\" arXiv 2024. \\\n[3] Ryali et. al., Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles, ICML 2023.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Strengths:\n\n- The proposed method is very well motivated. Reproducibility is high.\n\n\nWeaknesses:\n\n- Although there is a performance improvement, my primary concern is that the innovation of the paper is rather limited. The paper proposes an empirical method for combining multiple visual encoders in a multimodal model. The feature fusion operation is based on cross-attention and linear projection layers. Since ensemble methods are a fundamental concept in machine learning, the paper does not provide new insights, and considering the increased computational cost, the performance improvement is expected and not surprising.", "theoretical_claims": "The paper does not present any theoretical contributions. Instead, it employs an ensemble of multiple vision encoders to address the diverse tasks encountered in MLLMs.", "experimental_designs_or_analyses": "The authors conducted extensive experiments on combining visual encoders, covering both individual and joint usage scenarios. This exploration provides practitioners with insights into the effectiveness of different encoder combinations. Additionally, the authors analyzed the impact of various training stages in LLaVa-style video language model (VLM) training and offered observational conclusions on the importance of each stage in the two-stage training process.", "relation_to_broader_scientific_literature": "The authors enhanced the model's performance by integrating multiple visual encoders, an approach that is closely related to ensemble learning in machine learning.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Q1: What are the reasons behind the authors' selection of these four encoders? For example, the chosen video model ViViT was released in 2021. However, many more advanced methods, such as MViTv2 and VideoMAE v2, are now available.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "ZJnHllnwJ9", "reviewer": "Reviewer_44Cd", "summary": "The authors propose to use multiple video encoders (instead of a single encoder) for visual feature extraction in the context of video-LLMs. They propose a simple, straightforward technique to ensemble multiple encoders with a clever feature fusion strategy leading to lesser inference-time FLOPs than any of the individual encoders. Thorough and extensive evaluations establish the usefulness of proposed method while also uncovering some interesting insights on visual encoders for video tasks.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 44Cd, thank you for your time and effort on reviewing our work. We are grateful for the insights that you have provided us. Specifically, we appreciate that you have seen our work to have **novel and clever strategy** with **less inference-time and FLOPs**, providing **interesting insights** on visual encoders for the underexplored field of video understanding. We appreciate that you see the main weakness of the paper to be in clarification, which we will address in order below along with your question. \n\n**C1: On L125 (Right), why is $t$ the same across each $v_e$?** \\\nThis is because, prior to spatial alignment, we performed temporal alignment. You are right that video encoders may have temporal pooling, which alters the temporal resolution. Therefore, we achieve temporal alignment in the most reliable and straightforward way—by adjusting the number of input frames fed into each visual encoder. E.g., we feed 32 frames to ViViT so that the output $t=16$. \n\n**C2: Can you discuss motivation for your feature fusion design?** \\\nWe wanted a design which could adaptively and efficiently select the most relevant visual information from all of the encoders. The simplest methods, such as sequence and channel concat, allow us to naively combine the information at its full resolution, but are inefficient and limited, e.g., sequence concat ignores our spatio-temporal alignment and orders in sequence. We can make this more efficient by summing the features together with learnable scalar weights for each, i.e., the “Learnable W” row in Table 2(c). This reduces the output sequence length. However, this has much weaker expressiveness than other methods and is still not adaptive. \n\nCross-attention achieves the best of both worlds, being expressive while still being efficient. Using learnable queries as input, we adaptively set our encoder feature weights based on the input video and sum the features. The final output is of shape [BxLxD] instead of [Bx(NL)xD] where N is the number of encoders. Our ablations in Table 2(c) compare all of these feature fusion methods and find ours to be the most performant, along with channel concat. In fact, both methods are similar at heart. Channel concat creates a feature of shape [BxLx(ND)], but immediately has a linear layer following it to project into the LLM dimension of D. Our cross attention does a similar feature consolidation but constrained at the encoder level, i.e., each encoder’s feature gets one global weight. \n\n**C3: Can more output tokens from pre-fusion projector help more temporally complex tasks? Can the lower token count bottleneck MERV on Perception test and SSv2?**\n\n| | LanguageBind-LLM | MERV w/ 4 encoders |\n|:-:|:-----:|:-------:|\n|**Tkns**| Perception / SSv2 | Perception / SSv2 |\n|1|42.85/32.97|44.52/33.89|\n|4|43.31/33.34|43.16/35.81|\n|16|43.18/36.29|46.45/41.47|\n|64|44.43/37.36|46.21/**42.01**|\n|100|**45.56**/37.72|44.96/40.58|\n|144|43.94/**38.82**|45.86/39.40|\n|256|43.51/35.25|**47.26**/40.06|\n\nWe evaluated LanguageBind-LLM and MERV on PerceptionTest and SSv2 across varying pre-fusion output token counts. The results indicate that while more tokens can help with more temporally complex tasks, performance gains generally plateau beyond 64 tokens, suggesting that 64 tokens offer an optimal trade-off between performance and efficiency. We agree that limited token count could lead to an information bottleneck depending on the complexity of the task at hand, but in our studies, 64 tokens seems to be enough. \n\n**C4: In Figure 4, it may be unfair for MERV to compare with ViViT-LLM where ViViT-LLM uses more TFLOPs.**\n\nTo make the TFLOPs comparable, we perform an additional comparison where we use pre-fusion projector (pfp) with ViViT-LLM, so that the visual token size is now 64 per frame, instead of 196. \n\n|Model|TFLOPs|SSv2|SSv2-Temporal|\n|:---:|:----:|:--:|:-----------:|\n|MERV|17.19|**42.01**|36.84|\n|MERV (full)|17.19|39.76|**40.65**|\n|ViViT-LLM|27.12|26.78|39.77|\n|ViViT-pfp-LLM|13.00|29.07|35.15|\n\nAs expected from Table 2(a), our pre-fusion projector is designed so that it does not hurt the performance while using fewer visual tokens. Regardless, our method shows better or matching performance with single-encoder ViViT, with or without pre-fusion projector.\n\n**C5: What is the black outermost line of Figure 1 (Right)?** \\\nThe thin black line is the edge of the graph. Thank you for spotting them. We will fix the graph to be more clear.\n\n**Q1: Can we make the queries text-conditioned? Did we try this?** \\\nYes, we tried. As of now, current Video-LLaVA training datasets have vastly different text distribution with the benchmarks. As we are doing zero-shot evaluation, we did try and find that our feature fusion module does not generalize to such out-of-distribution textual prompts when the queries fed to the fusion module. We think this could be an exciting scope for future research to effectively generalize a text-conditioning which works on broader examples.", "claims_and_evidence": "Yes. Extensive experimentation demonstrates usefulness of proposed method.", "methods_and_evaluation_criteria": "Yes. Clear method with extensive evaluation.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "Authors follow standard experimental settings on established benchmarks.", "relation_to_broader_scientific_literature": "Visual encoders are an important component in multi-modal LLMs (especially) video, and is relatively under explored. The authors investigate a novel direction of ensembling multiple visual encoders with negligible increase in runtime costs. This direction can be valuable to the video understanding community, especially for video QnA.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**\n1. The paper is well written with clear presentation of methodology and experiments. \n2. The authors explore a timely topic (visual encoders for video) that could be valuable to the video understanding community. \n3. A simple and straightforward modification to existing baselines with clever design choices to maintain inference compute. \n4. Extensive experimentation achieving strong results improvements across a range of video QnA benchmarks. \n\n\n**Weaknesses**\n1. Minor improvements for better clarity (see comments / suggestions).", "other_comments_or_suggestions": "1. On L125 (right column): \"As t is the same across each v_e\" - why is this? do the video encoders used have no pooling / compression across time axis? \n2.  Eq (2) for feature fusion: can you discuss motivation for this exact design choice?\n3. Table 2 (b) - could these numbers be reported (maybe in appendix) for Perception test and SSv2? \n    * Could more tokens help with more temporally complex tasks? \n    * This is motivated by ViViT outperforming MERV on several such tasks. Could the lower token count (information bottleneck) of MERV be a reason for that?\n4. The SSv2 analysis against ViViT, maybe repeat with same token count for MERV? It maybe unfair for MERV to compete against a more expensive ViViT version that uses more TFLOPs. \n5. What is the black outer-most line in Figure 1 (right)? This is unclear.", "questions_for_authors": "1. Could the feature aggregation be conditioned on the textual prompt? Is this a direction that was explored at any point?\n    * E.g. In feature fusion cross-attention operation, the queries could be derived / conditioned on from textual prompt.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZuYgmpFJaY", "reviewer": "Reviewer_jZRP", "summary": "The paper presents a VideoLLM that aligns and unifies the outputs of four different visual encoders to improve its video understanding capabilities. Specifically, the authors use a spatial encoder (DINOv2), a temporal encoder (ViViT), an image-language encoder (SigLIP), and a video-language encoder (LanguageBind). They design the inputs to the four encoders to be spatio-temporally aligned and train a pre-fusion projection network followed by a cross-attention network to respectively transform the features from the different encoders into a common space and then combine those features. The authors perform relevant experiments to evaluate their approach on multiple video Q&A datasets and compare against baseline methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer jZRP, we thank you for spending your time and effort on reviewing our work. We appreciate your recognition of our thorough experimental setup and ablations that sufficiently backs up our claim, as well as its relevant and timely contribution towards a broader collection of works regarding video understanding models. We will address your two main questions, as well as the two other comments you had for our work.\n\n**Q1: Total running time and FLOPs of compared methods?** \\\nWe obtained the running time and FLOPs for the methods in Table 1. In the main paper, we have calculated FLOPs by fixing the length of the textual tokens. But due to different generation schemes, this method cannot be applied to some VLMs.\n \nTo make a fair comparison between VLMs, we use the same video and with the prompt \"Describe this video.\" We follow the prompt builder and the video-processor provided by the official implementation. We report the Time-to-first-token (TTFT) as a metric for running time. Our FLOPs were measured in the same TTFT setup.\n\n|  | TTFT (Inference) | TFLOPs |\n|---------------|------------------|--------|\n| Video-Chat\t| 135ms| 3.22   |\n| LLaMA-Adapter | 124ms| 2.69   |\n| Video-LLaMA   | 169ms| 11.07  |\n| Video-ChatGPT | 321ms| 10.69  |\n| SeViLA^  | X (two LMs)  \t| X  \t|\n| LLaMA-VID-7B  | 217ms | 3.05   |\n| LLaMA-VID-13B | 242ms| 3.47   |\n| Video-LLaVA   | 312ms | 15.93  |\n| MERV**  | 261ms | 10.5   |\n\n^ Since SeViLA is MCQ-only VLM, needing to run LLM per each MCQ option, we do not compute TTFT and FLOPs as we cannot compare them with other models in a fair manner. \\\n** As we are using a different prompt, the FLOP value is different from the main paper.\n\nOur method, despite having 4 encoders, has far fewer FLOPs compared with Video-LLaVA (which our work primarily builds on), due to the compressed visual representation used. While LLaMA-VID achieves fewer FLOPs by restricting a single frame to have 2 visual tokens, we can achieve a much better performance with a similar step time with our efficient implementation. \n\n**Q2: Can accuracy be improved by using longer input sequences to the pre-fusion projector, and how would it affect the computational overhead of feature fusion?**\n\nThis is a very interesting question! Below we tried using inputs with longer sequences to the pre-fusion projector by scaling up temporal resolution (as spatial resolution is fixed at 224x224) to see if our method improves.\n|\tMERV   |  TFLOPs |  MSVD | MSVD-Score | MSRVTT | MSRVTT-Score |  TGIF | TGIF-Score | Perception |\n|:---------:|:------:|:-----:|:----------:|:------:|:------------:|:-----:|:----------:|:----------:|\n| 16 frames | 17.19 | **61.61** |\t**3.41**\t|  **45.17** | \t**2.75** \t| **46.11** |\t**2.65**\t|\t46.21   |\n| 32 frames | 26.23 | 61.28 |\t3.40\t|  44.75 | \t2.74 \t| 45.42 |\t2.62\t|\t**47.77**   |\n* _As gpt-3.5-turbo-0613 is now deprecated, we use GPT-4o-mini for open-ended evaluation._\n\nWe find that our method fails to take advantage of the extra temporal resolution for most datasets except for the Perception Test (which is a temporally challenging dataset). This indicates an upper limit to the capacity of our LLM for processing extra visual information, in which case maybe extra token selections or compression needs to happen. \n\nRegarding the second part of the question, increasing the input sequence length (i.e., larger $l$) to the pre-fusion projector leads to a linear increase in computational overhead for the feature fusion step, as can be seen from Eq. 2. However, for the subsequent LLM processing, the overhead grows quadratically with sequence length.\n\n**C1: Looking at failure modes when removing DINOv2 and ViViT**\n\nThank you for the suggestion, we really appreciate it! We have not thought of the qualitative results of visualizing the failure cases when removing ViViT from MERV. We hypothesize that the failed cases are the videos that contain a lot of temporal movements, e.g., rapid camera movement. The hypothesis comes from a similar experiment, Figure 10 in Appendix, where we look at the attention weight of the feature fusion module, and see which video gains the highest weight for each encoder. As we saw that it was often the video that has large movements that is preferred by ViViT, we believe that these videos will have the most impact when ViViT is removed from MERV. We will add the suggested analyses in the camera-ready version of the paper.  \n\n**C2: What is \"temporally indistinguishable classes\" in Something-Something V2 dataset?**\n\nYes, your interpretation is exactly right. In particular, we selected action classes where the temporal reverse of one class approximates another class — as you pointed out, reversing frames of \"Pulling from Left to Right\" approximates frames of \"Pushing from Right to Left\". We identified 12 such class pairs from SSv2, which are listed in the appendix. We understand that our original explanation could be clearer, and we'll revise the paper following your helpful suggestion!", "claims_and_evidence": "The central claim -- that of combining different visual encoders focusing on different aspects of image, video, and their correlations with text inputs to improve video understanding performance -- is sufficiently backed up by experimental results. However, the additional benefits of computational efficiency can be evidenced further:\n\n1. In Table 1, what are the total running times and FLOPs of the compared methods? How much does the proposed method improve on this aspect?\n\n2. Since the computational improvements reported in Fig. 4a are 3-7 orders of magnitude higher than the FLOPs required for the pre-fusion projectors (Table 2a), can the accuracy numbers be improved further (while keeping the computational increases insignificant) by considering longer sequences as inputs to the pre-fusion projectors? Or would longer sequences also significantly impact the computational overheads of the feature fusion step (Table 2c)?", "methods_and_evaluation_criteria": "The proposed method is technically sound, and the evaluation criteria are appropriate.", "theoretical_claims": "Not applicable - the paper presents experimental findings to justify the proposed approach.", "experimental_designs_or_analyses": "The experiments, including the ablations, are sound, and the analyses are generally thorough. One observation from the results (Fig. 5 in particular) is that the proposed approach improves significantly over DINOv2 and ViViT across the board, but marginally over SigLIP and LanguageBind in many categories. To complement these results, looking at failure cases due to removing individual encoders (especially DINOv2 and ViViT) can further help understand their specific contributions.", "relation_to_broader_scientific_literature": "With the development of language models for video understanding, the paper's contribution is relevant and timely, and it establishes new baselines for future video understanding models. It will likely interest the broader scientific communities working on language models, video understanding, and their intersections.", "essential_references_not_discussed": "While not a domain expert, I did not find any major missing references in my search.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "I am unclear on the authors' example of an action that is indistinguishable from itself if temporally reversed. Pulling from left to right is temporally distinct from pulling from right to left as (a) the hand location relative to the object is different, and (b) the direction of movement is reversed (Fig. 8, columns 2 and 3). However, pulling from left to right, if temporally reversed, can be indistinguishable from *pushing* from right to left. Was that the intended meaning of \"indistinguishable\"? Maybe the authors can make this point fully clear with a pair of temporally indistinguishable actions in Fig. 8?", "questions_for_authors": "Please refer to the comments in previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The manuscript received ratings of 5, 2, 4, and 2. Reviewers appreciated that the proposed method is well motivated and effectively enhances the capabilities of existing MLLMs following reasonable experimental protocols. Reviewers also raised questions including, limited technical novelty and the increased computational cost. Authors provided rebuttal to address reviewers concerns, especially improvements using the proposed approach without significant increase in the inference-time and FLOPs. Post-rebuttal, two reviewers remained positive whereas the other two persisted with their respective concerns. Given the reviews and rebuttal, the AC agrees with reviewer 44Cd and jZRP that the proposed approach is simple and is sufficiently backed up by experimental results in terms of performance gains without any significant increase in runtime costs. The simplicity and effectiveness of the proposed approach is also acknowledged by the other two reviewers (1K5Z and ukCZ). The reviewer 1K5Z further mentions that the reproducibility of the proposed aapproach is high and that the experimental exploration provides practitioners with insights into the effectiveness of different encoder combinations. The AC agrees with reviewers comments that the proposed study will be interesting for practitionars as it provides insights on building a more performant ensemble of multiple visual encoders which is computationally inexpensive at inference  due to the introduced fusion scheme, thereby being valuable to the video understanding community.
Decision: Accept (poster)