SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understanding
Authors: Mingze Xu
Mingfei Gao
Shiyu Li
Jiasen Lu
Zhe Gan
Zhengfeng Lai
Meng Cao
Kai Kang
Yinfei Yang
Afshin Dehghan
Venue: COLM 2025 (2025)
Keywords: Multimodal LLM
Video Understanding
Video Question Answering
Long-Form Video Understanding
TL;DR: We introduce SlowFast-LLaVA-1.5, a family of video large language models offering a token-efficient solution for long-form video understanding.

Abstract: We introduce SlowFast-LLaVA-1.5 (abbreviated as SF-LLaVA-1.5), a family of video large language models (LLMs) offering a token-efficient solution for long-form video understanding. We incorporate the two-stream SlowFast mechanism into a streamlined training pipeline, and perform joint video-image training on a carefully curated data mixture of only publicly available datasets. Our primary focus is on highly efficient model scales (1B and 3B), demonstrating that even relatively small Video LLMs can achieve state-of-the-art performance on video understanding, meeting the demand for mobile-friendly models. Experimental results demonstrate that SF-LLaVA-1.5 achieves superior performance on a wide range of video and image tasks, with robust results at all model sizes (ranging from 1B to 7B). Notably, SF-LLaVA-1.5 achieves state-of-the-art results in long-form video understanding (e.g., LongVideoBench and MLVU) and excels at small scales across various video benchmarks.

Published as a conference paper at COLM 2025
Hew Fas--LLaVA-11.. A Family of Token-Efficient Video Large Language Models for Long-Form Video Understanding
Mingze Xu", Mingfei Gao", Shiyu Li", Jiasen Lu, Zhe Gan, Zhengfeng Lai, Meng Cao, Kai Kang, Yinfei Yang", Afshin Dehghan
Apple
(mingze.xu2, shiyu.li, shinu.li.yinfeiy.. First authors; "Senior authors
Abstract
We introduce SlowFast-LLaVA-1.5 (abbreviated as SF-LLaVA-1.5), family of video large language models (LLMs) offering a token-efficient solution for long-form video understanding. We incorporate the two-stream Slow- Fast mechanism into a streamlined training pipeline, and perform joint video-image training on carefully curated data mixture of only publicly available datasets. Our primary focus is on highly efficient model scales (1B and 3B), demonstrating that even relatively small Video LLMs can achieve state-of-the-ee performance on video understanding, meeting the demand for mobile-friendly models. Experimental results demonstrate that SF-LLaVA- achieves superior performance on a wide range of video and image tasks, with robust results at all model sizes (ranging from 1B to 7B). Notably, SF-LLaVA-1.5 achieves state-of-the-art results in long-form video understanding (egg,, NongVideoBench and MLVU) and excels small scales across various video benchmarks.
1 Introduction
Video large language models (LLMs) (Maaz et al., 2024b; Lin et al., 2023a; Xu et al., 2024a) integrate video perception into pre-trained LLMs to process videos and generate responses to user commands. Although significant progress has been made, notable limitations remain in existing Video LLMs. First, they enhance perception and reasoning over long video sequences by leveraging the LLM's increasing context length and handling massive input frames (Shen et al., 2024; Chen et al., 2024d; Zhang al., 2024c). However, the potential for transferring this capability to highly efficient models is underexplored. Second, achieving optimal performance typically requires internal datasets and a complex training lifecycle, with selective parameters frozen at each stage (Li et al., 2024a; Zhang et al., 2025a). These intricate designs lead to high computational costs and reproducibility challenges. Third, many Video L.MSs (Zohar et al., 2024; Li et al., 2024d) are optimized exclusively for video tasks, limiting their effectiveness as joint models for image understanding tasks.
Building upon the success of SlowFast-LLava (Xu et al., 2024b), introduce SlowFast- VA-115, a new family of Video LLMs for long-form video understanding, focusing on the most efficient model scales (1B and 3B). Our model family is both effective and token-efficient in modeling long-range temporal context. This is achieved by employing the SlowFast mechanism, which balances the trade-off between processing more input frames that significantly increases the token count and computational cost, and reducing tokens per frame that inevitably loses fine-grained details. Specifically, the Slow pathway captures detailed spatial features at a low frame rate, while the Fast pathway operates at a high frame rate with fewer tokens per frame to focus on motion cues. The success our model also relies on streamlined training pipeline and a carefully curated data mixture. Our model training consists of only two stages. The first stage is supervised fine-tuning on image-only data, providing good foundation for general knowledge and reasoning. The second stage conducts video-image joint training to learn spatial and temporal features for video understanding while maintaining strong performance in image understanding. To
1

---

Published as a conference paper at COLM 2025
ensure seamless reproducibility, all pre-trained weights and training datasets used in this work are publicly accessible.
We comprehensively evaluate models various video and image benchmarks. Experi- mental results demonstrate that SlowFast-LLaVA-155 achieves state-of-the-an performance in long-form video understanding. Notably, our 7B model scores 62.5% on LongVideeBench and 71.5% on MLVU, outperforming existing methods by a clear margin. SlowFast-LLaVA, 1.5 also excels at smaller model sizes, achieving 56.6% and 60.8% on Video-MME (w/o sub) at the 1B and 3B scales, respectively. As a unified image and video model, it maintains strong image performance despite the simple training recipe.
Our main contributions are as follows. First, we introduce SlowFast-LLaV--15, new family of Video LLMs ranging from 1B to 7B parameters. We demonstrate the effectiveness of incorporating the SlowFast mechanism into a supervised fine-tuning framework, modeling long-range context while maintaining high efficiency. Second, our model family provides enhanced reproducibility by using only two training stages and publicly available datasets, distinguishing it from existing methods. Third, SlowFast-LLLVA-I the-art performance on long-form video understanding. Moreover, our smaller models (1B and 3B) clearly outperform comparable Video LLMs across video benchmarks.
achieves the sfate-of-
2 Related Work
Image Large Language Models have gained widespread attention (Achiam et al., 2023; Team et al., 2023; Touvron et al., 2023; Chen et al., 2024e; Bai et al., 2025). Significant progress across multiple fronts includes: (i) enhancing data quantity and quality during pre-training (McKinzie et al., 2024; Liu et al., 2024a; Lin et al., 2023b; Li et al., 2024c) and supervised fine-tuning (SFT) (Zhang et al., 2025b; Deitke et al., 2024; Chen et al., 2024a; Wang et al., 2023; Tong et al., 2025); (ii) accommodating images of various high resolutions (Lin et al., 2023c; Zhang et al., 2024b; Wang et al., 2024b); (ii) improving architecture designs, including different visual encoders (Zhai et al., 2023; Tong et al., 2024; Shi et al., 2024) and lision-language connectors (Li et al., 2023a; Cha et al., 2024); and wv) conducting comprehensive studies for easy deployment (Team et al., 2023; Marafioti et al., 2025). These rapid advancements also establish a strong foundation for related areas such as video understanding (Maaz et al., 2024b; Lin et al., 2023a), referring & grounding (You et al., 2023; 2024), and visual agents (Durante et al., 2024; Yang et al., 2025).
Video Large Language Models have become an active research area (Li et al., 2023b; Song et al., 2024; Chen et al., 2024b; Zhang et al., 2024e; Zohar et al., 2024). Early Video LLMs are developed as specialist models (Zhang et al., 2023; Cheng et al., 2024; Xu et al., 2024a; Ryoo et al., 2024), achieving strong performance on video tasks but with some trade-offs in image understanding. Training-fie Video LLMs (Kim et al., 2024; Xu al., 2024b) offer an efficient alternative by leveraging Image LLMs without fine-tuning on video data, enabling flexible deployment across various applications. Recent models (Zhang et al., 2024g; Liu et al., 2025; Zhang et al., 2025a) are jointly trained on video and image datasets, obtaining superior results in both modalities. Long-form video understanding (Zhou et al., 2024; Wu et al., 2025) gained increasing attentions, addressing hour-long videos (Chen et al., 2024d; Li et al., 2024d) or live streams (Qian et al., 2024; Zhang et al., 2024a) while optimizing the token efficiency (Lee et al., 2024b). The proposed SlowFast-LLaVA-1.55 is family of Video LLMs designed for modeling long-range temporal context. It enhances SlowFast-LLava, (Xu et al., 2024b) by implementing the SlowFast design within a unified video-image training framework, achieving state-of-the-art performance with efficient token utilization.
3 SlowFast-LLaVA-1.
We provide a detailed explanation of SlowFast-LLaVA-15 (abbreviated as SF-LLaVA-155), which incorporates the Flow video projector into a LLL architecture, improv- ing long-range temporal modeling while optimizing token efficiency. In contrast to its training-free pioneer (Xu al., 2024b), this paper (i) systematically investigates different in- stantiations based on the generic FlowFast idea (Sec. 3.1), (ii) designs compact yet effective training pipeline (Sec. 3.2), and (iit) introduces tailored data mixtures using only publicly available datasets for each training stage (Sec. 3.3).
2

---

Published as a conference paper at COLM 2025
FPS Uniform Sampler
Slow Pathway
+0000
Visual Encoder
Flatten& & Concal
Projection
Pre-trained
Input
<epp
p00r
Large Language
Video
Model
4. titte
pistt
Fast Pathway
Text
Figure Visualization of the video understanding pipeline in SlowFast-LLaVA-113 Com- pared to its training-free pioneer (Xu et al., 2024b), our projector and LLM are fine-tuned throughout the training cycle, while keeping the vision encoder frozen.
3.1 Model Architecture
As shown in Fig. 1, the architecture of SF-LLaVA-115 follows the core design principle of SF-LLaVA (Xu et al., 2024b). It takes a video /image and a question Q as inputs and responds with textual answer A. For video inputs, sample N frames, = {4, 12, ..., IN). at a fixed frame rate without special frame assembling (N equals 1 for image input). After that, visual encoder (eg., OryxViT (Liu et al., 2025)) is used to extract frame-level features F, R NxH from the inputs independently, keeping their original aspect ratio. The video and image feature tokens are then fed into different projectors, with video using the two-stream FlowFast projector and image using a two-layer MLP.
The SlowFast projector processes Fy through two pathways, one dedicated to capturing spatial patterns and the other modeling motion cues.
The Slow pathway, which focuses on capturing detailed spatial semantics, operates at reduced frame rate by downsampling the total frame count from N to Nslow To further improve the efficiency while preserving sufficient details, it applies spatial pooling over
F, with proper strides of o6 x dw- The output feature is Fslow R x1fslow x Wyslow where Hslow H/or, and Wsloow W/
The Fast Pathway, which focuses modeling long-range context, maintains the original frame rate, while downsampling more aggressively on the spatial resolution to Hfast Wfast, The output feature is plast E N x where N N, H < Hslow and Wffttt Wsloow
Falow and plast are flattened and concatenated together as a token vector Fa as the final visual input to the LLM. A dedicated special token is typically used to separate Felow and Ffast, assisting the LLM in distinguishing the two sets of features.
which serves
3.1.1 Instantiations of SlowFast
Next, we describe two approaches for organizing the Slow and Fast tokens.
The Group-based SlowFast (GSF) places the Slow tokens before the Fast tokens (Appendix Fig. 2 above). This design is inspired by the AnyRes (Zhang et al., 2024f) technique in image understanding, where the Fast tokens provide a global overview of the video and the Slow tokens capture fine-grained spatial details. Notably, SlowFast- LLaVA (Xu et al., 2024b) works effectively only under this setting, as it is training model that benefits from "overfitting" to its image backbone (i.e., LLaVA-NeXT),
The Interleaved SlowFast (ISF) arranges the tokens according to their spatial and temporal order (Appendix Fig. 2 below). Since Slow and Fast frames contain different numbers of tokens, a learnable special token is utilized to separate adjacent frames, allowing the LLM to distinguish which frame a token belongs to. Different from GSF, Nfat equals to N Nolow in this approach. ISF balances the presence of both token
3

---

Published as a conference paper at COLM 2025
types throughout the input sequence, preventing the model from becoming overly focused on one type of information at a time.
Unless noted otherwise, use GSF by default, as it aligns better with the image pipeline using AnyRes inputs. Interestingly, experiments (Sec. 4.4.1) show that SF-LLaVA-115 is not sensitive to this setting, suggesting that the generic SlowFast idea and our training recipe are the main reason for the strong performance on long-form video understanding.
3.2 Training Pipeline
The training pipeline of SF-LLaVA-1.5 is much simpler than most of the existing Video LLMs (Chen et al., 2024e; Zohar et al., 2024; Zhang et al., 2025a; Li et al., 2024d) with only two training stages, detailed in Table
Settings
Stage
Stage II Image Video Projector & LLM
Dataset
Image
Trainable
Projector LLM MLP w/ GELU
Image Projector
MLP w/ GELU
Video Projector
SlowFast
Batch Size
512
512
Learning Rate
2
2
Context Length
8K
16K
Number of Input Frames
1
1 or 128
Max Image Resolution
1280 x 1280
1536 x 1536
Max Video Resolution
480 x 480
Training Steps
epoch
1 epoch
Table Training settings for SlowFast-LLaVA-115
Stage (image understanding) conducts SFT with images to provide a good warmup status for video understanding. For simplicity and efficiency, we do not any extra pretraining stages (Li et al., 2024a) or image splitting strategy (Lin et al., 2023c), although they have proven to be effective for boosting text-rich results. Instead, we use native resolution inputs following Oryx (Liu et al., 2025), where, for each image I; E R we have low resolution wfo and high resolution flishh input. The low-resolution image is obtained by simply resizing the original image base resolution, as in Hlo = resize House x He high
high
we keep its original aspect ratio and resize it to H Hligh
x Whish as in Eq. and 2,
For 1
v VOl"! /H, W,), H. X W; g0
scale
V 01mi /(H, W:),
if x W; M
(1)
1.0,
otherwise,
Hhiihh int(H; * scale/p) *p
(2)
whishhh int(W; scale/p) #
where H. and whish represent the resized heights and widths and p denotes the patch size of the ViT-based vision encoder. Eq. 1 calculates the resizing scale ensuring that the area
of 1, high is between two pre-defined, minimum area 0m and maximum area 0 paax thresholds.
Eq. makes sure that both H.iish and whithhh are multiples of p. To accommodate different input resolutions, the original position embeddings of the vision encoder are rescaled using bilinear interpolation. After feature projection, the low-resolution and high-resolution image features are concatenated together as the final image feature.
Stage II (joint image & video understanding) performs SFT jointly with images and videos, initialized by the pre-trained checkpoint from Stage I. By default, we keep the image resizing
setting the same Stage I, except that increase the maximum area threshold, @ larger value for better performance. For video, each frame uses a single resolution that is
set using the same strategy as in Eq. and 2, where we use ! and 8 Vmmex to denote the corresponding minimum and maximum area thresholds.
4

---

Published as a conference paper at COLM 2025
3.3 Data Mixture
Our image and video mixtures are detailed in Table 8. Many state-of-the-art models (Li et al., 2024a; Zhang et al., 2025a) achieve superior performance using internal training data that is unavailable to the research community. To ensure the reproducibility of our models, we only include publicly available datasets in our data mixtures.
Image Mixture. General, TextRich, and Knowledge are fundamental for developing the reasoning capabilitiees of multimodal LLM, which can ultimately benefit both image and video understanding. We begin with datasets from these three categories in MM1.5 (Zhang et al., 2025b) and evaluate additional datasets for each group from LLaVA-OneVisioe et al., 2024a) and Intern VI.2.5 (Chen et al., 2024e). Datasets are included in our mixture only if they empirically improve performance. The final mixture contains 4.67M samples.
(Li
Video Mixture. We build diverse set of video instruction-following datallss datasets. We begin with LLaVA-Hound (Zhang et al., 2024e), ShareGIT4Video (Chen al., 2024b), VideoChatGPT- Plus (Maaz et al., 2024a), and ActivityNet-QA (Yu et al., 2019) to include large-scale video data with caption and QA labels. We add NExT-QA (Xiao et al., 2021) and Perception Test (Päträucean et al., 2023) to improve performance on temporal reasoning. Furthermore, (Zhang et al., 2024g) and Cinepile (Rawal et al., 2024) to
we incorporate LLAV-Viido-122 enhance long-form video understanding. Finally, we filter out duplicate videos from the same data source and construct our final mixture with 2.01M training samples.
4 Experiments
We evaluate SF-LLaVA-1.5 across multiple video and image QA benchmarks (details will be provided in Appendix A.2). For video, we focus on long-form video understanding, while also reporting the results in general video QA and temporal reasoning. For image, we evaluate the models from general, knowledge, and text-rich perspectives.
4.1 Implementation Details
Model Architecture. We use Oryx-ViT' (Liu et al., 2025) with patch size 16 as visual encoder and Qwen2.52 (Bai et al., 2025) series of LLMs at varying scales as the backbone. We employ different projectors for video and image inputs. Specifically, the Group-baset SlowFast (GSF) structure is used to aggregate video tokens. For the Slow pathway, we uniformly select Nslow 32 frames and apply 2 x 2 pooling to their extracted features. For the Fast pathway, use features of all frames (i.e., N N 128) and downsample their features to 4 x 4 tokens. For the image projector, we use two-layer MLP with GELU activation function.
Training Details. As summarized in Table 1, we freeze the visual encoder in all stages and only fine-tune the projectors and LLM. We use the same hyperparameters for 1B, 3B, and 7B models, setting the total batch size to 512 and learning rate to -5, All models are trained 128 H100-80G GPUs for 1 epoch.
Training Stage I only uses image understanding data. The low resolution image 1!o" is fixed at Hhuse x Whise = 384 x 384 and the high resolution image r is obtained as in Eq. and Eq. 2, where 0!t!in = 0 and 0! = 1280 The maximum context length is set to 8K. The models trained by this stage are named as SF-LLaVA-1.5-mmaa
Training Stage II continues training based on SF-LLaVA-1.5-mman by combining our video and image data mixture. For image, the high-resolution image is obtained in the same way as Stage except that we increase g g to 15362. For video, we follow prior work (Zohar et al., 2024) and sample frames at 1 FPS. We set the max frame number to 128 and uniformly sample the frames if the number exceeds this upper bound. For
each video frame, we set 6 V 2882 and g V = 4802. The maximum context length set to 16K. The models trained by this stage are named SF-LLa
hhttps//hhugginggacco///HHU ViT Phtp:s//tugggngfaeeco//vvee Qwen.

---

Published as a conference paper at COLM 2025
Max Input
Max Inpart Tokers
Long-Form Video Understanding
Temporal Reasoning TempComp (mc)
Model
VideoMME (w/osub)
PencepTsst (val)
LongVideeoBecc (vai)
LVBench (avg)
NENT-QA
Promes
(m-ang)
(test)
Compurisne
LLaVA-OV-058 (1i al, 2024a)
32
6K
44.0
49.2
45.8
50.3
32.7!
53.2
57.2
MMLL5-1 (Zhang et. 20256)
21
3K
15.7
13.9
71.8
LinVI-Mipha 168 (Gáo et al, 2024)
120
44.5
49.7
56.2
45.2
71.1
Apollo-1.58 1.58 (Zoher et 2024)
25ps
3K
53.0
61.0
54.1
63.3
60.8
IntemVL22522
et 202te)
16K
77.2"
Qwen2-VL-228 (aag 20246)
2ips
16K
55.6
53.9
48.7"
62.77
39.4
61.6"
77.2'
SF-LEaVA-15-1B
128
9K
56.6
61.9
54.3
64.3
39.7
60.5
76.7
58 Madi' Comparison
VILAI15-30 (Liu 2024c)
8
2K
42.2
49.1
42.9
44.4
56.1
MM1.5-38 (2hang al, 20256)
24
3K
49.5
45.4
74.7
LongVU-3320 (Shen al. 2024)
1ips
8K
51.5
55.9
I1lemnV122-4B al, 2124()
64
53.9"
LinVT-Bin3-48 (Zohar et 2024)
120
58
36.6
80.1
Apollo-3i (Zohar 2024)
25ps
3X
58.4
65.0
55.1
68.7
62.5
SF-11 15-38
128
9K
60.8
65.8
57.3
68.8
43.3
64.0
80.8
79 Made' Comparisov
MML557B (Zhagg et al, 2025))
24
3K
53.5
49.4
26.9
Kangano-88 (1 (liuet 2024b)
64
10K
56.0
54.8
61.0
39.4
62.5
Orvd 5-78
58.87
LVVV at.
MLaVA-Video-7 (2hang 20344g)
64
11K
83.2
Apolle-B (Zohar
2024)
25ps
2X
613
67.3
58.5
20.9
61.9
NVILA-8B 8B (Liu et al, 2024c)
256
8K
64.2
65.4
57.7
20.1
44.0"
69.7"
82.2
Inter 21..5...55588 (Chen 2024e)
16K
69.0
85.07
Owen2 VL-7B (Wang al,, 20144))
2/ps
16K
63.3
62.3
53.6
69.8
44.7"
67.9*
81.2"
SF-LLaVA-15-76
128
9K
83.0
69.6
62.5
71.5
45.3
68.8
833
Table 2: Comparison with state-of-the-art models on video understanding. Bold and
underlined are the best and second-best results for each task. *denotes reproduced results.
4.2 Video Understanding Results
We mainly compare SF-LLaVA-1.5 with state-of-the-art Video LLMs that are trained on publicly available datasets. Here we highlight some key observations based on Table
First, VA-11 achieves state-of-the-art results in long-form video understanding. Specifically, SF-LLaVA-1.5 outperforms existing models on both LongVideoBenee and LVBench across all model sizes. For reference, it surpasses Intern VL.2.5 at both 1B (+2.3% on LongVideo
and +1.8% on LVBench) and 7B (+2.5% on LongVideoBende and +2.1% on LVBench) scales. SF-LLaVA-1.5 also exhibits leading performance on MLVU. Compared to the state-of-the-art model, Apollo, achieves +1.0% at the 1B scale and comparable results at other scales. Additionally, SF-LLaVA-1.5 delivers better results even compared to Video LLMs tailored for long videos, such as LongVU. For instance, SF-LLaVA-1.5-31 significantly surpasses LongVU-3.2B by +9.3% on Video-MME. and +12.9% on MIVU.
Second, SF-LLaVA-1.5 is the state-of-the-art model at the smaller scales. As edge deploy- ment becomes increasingly important, more models are emerging in the 1B and 3B sizes, including LLaVA-OV, Intern Qwen2-VL, and Apollo. For reference, 3F-LLaVA-115-1 surpasses Qwen2-VL-2B across benchmarks (e-g., 56.6% 55.6% on Video-MME, 61.9% vs. 53.9% on Perception Test, 64.3% vs. 62.7% on MLVU), Compared to Apollo-1.5B, SF- LLaVA-1.5-1B exhibits a +3.6% improvement on Video-MME, while leading other tasks. Similarly, at the 3B scale, SF-LLAVA-115-0 outperforms Apollo-3B by +2.4% on Video-MME for general Video QA and by +1.5% on TempCompass for temporal reasoning.
Third, SF-LLaVA-115 optimizes the trade-off between performance and efficiency. SF- LLaVA-1.5 excels in long-form video understanding while using fewer tokens than existing methods. Using Oryx1.5 as an example, SF-LLaVA-115 utilizes only ~65% of its input tokens (9K vs. 14K) but processes twice as many frames (128 vs. 64), resulting performance on nearly all benchmarks (e.g,, 63.9% 58.8% Video-MME and 71.5% vs. 67.5% on MLVUU. Notably, NVILA uses a similar number of input tokens as SF-LLa SF-LL1VA-115, yet SF-LLaVA-1.5 surpasses it by +4.8% on Ling and +1.4% on MLVU. These results demonstrate the advantages of SF-LLaVA-1.55 in modeling long-range context.
better
Fourth, SF-LLaVA-115 exhibits robustness across tasks and model sizes. SF-LLaVA-1.5 con- sistently achieves strong performance across all benchmarks in Table 2. This demonstrates two key points: i) using two-stream SlowFast inputs is beneficial for modeling long-range

---

Published as a conference paper at COLM 2025
Knowledge
TextRich TeavQQ (val)
Model
Input
Stage 4
SQJ
MMMU
MathV (lestmini)
Met-Vet RW-QA
RRBeecch (test)
DocVQA
Pivell
(test)
(test)
(val)
(test)
28 Mostei Comparison
Gemini Nano-1 [Team et al, 2023)
37.9
26.3
27.3
62.5
72.2
LLVV--V--..5 (Li 2124a
5.31M
57.1
67.2
31.4
34.8
55.6
70.0
MM15--88 (Zhang 21250)
452M
72.5
InternV1.2.5-18 Chen
(2004e)
9.3M
43.2
57.5
72.0
84.8
MolmoP- (Deltke et al., 2024)
4.10M
$6.4
34.9
34.0
60.4
78.8
77.7
FF-LLaVA-1,5-mmaae
2.36M
20.8
87.8
39.3
51.2
41.1
57.1
69.5
70.2
85.2
VH-1LaVA-1.--18
2.66
2
72.8
87.7
4035
51.00
51.2
99.2
70.0
71.3
854
3B Madei Comparison
(Team
65.9
74.3
MiniCPM-V2-3B (Yao 2024)
1.8M
80.7
38.7
74.1
71.9
BLPP3-4B (Xue al, 2024)
88.3
41.1
39.6
60.5
71.0
MM15--38 (Zhang et al, 20258)
4.52M
65.7
85.8
37.1
+4.4
41.0
56.9
65.7
76.5
87.7
Phi-3.5-V-48 (Abdin
2024)
78.1
91.3
43.0
43.9
72.0
SF-LLaVA-1.5 a...aaaV-..mmmmm Immagg---
2.36M
90.0
57.0
72.0
87.5
SF-LLaVA-15-338
7B Model Comparison
V1--A1.--B in al,, 20236)
26.6
38.6
36.7
52.7
68.5
40.6
Idefix2-88 (Laurengon onee atall 2024a)
2.95M
3
43.0
51.4
73.0
74.0
Cambrian- 1-88 Tong ol. 2025)
73.0
80.4
42.7
49.0
64.2
62.4
71,7
77.8
LVVVA-OV-28
2024a)
5.31M
63.2
57.5
87.5
MML15-78 (Zhang et 20250)
Oryx1 5-78
2025)
2.36M
79.7
47.1
71.3
75.7
9011
IntemnV1.2.5-881 (Chem et al, 2024e)
9.63
2
84.5
56.0
64.4
70.1
79.1
93D
Qwen2-VI. VL.78 (Wang et. all, 20244))
3
83.0
54.1
58.2
62.0
71.1
84.3
94.5
SF-L 1L-LaVAA----Iuge-- VH-11aVA-1.5-77
2.36M
79.2
91.8
47.0
61.0
50.1
64.6
742
75,4
897
Table 3: Comparison with state-of-the-art models on image understanding. This table denotes "MathV" for Math Vista and "RW-QA" for RealWorldQA. Bold and underlined are the best and second-best results for each task.
temporal context across various video tasks, and ii) our proposed training pipeline and data mixture enable seamless generalization from mobile-friendly to large-scale Video LLMs.
4.3 Image Understanding Results
We also compare SF-LLaVA-1.5 against recent multimodal LLMs on image understanding, as shown in Table 3, highlighting the following observations.
First, SF-LLaVA-1.5 excels smaller model scales. Similar to video, SF-LLaVA-1.5's 1B and 3B models achieve competitive results across image benchmarks. Specifically, SF-LLaVA-115 by +3.5% on A12D and +7.8% on Math even though we use less than 30% of their input resolution. When compared to MolmoE-1B, our model clearly wins on MMMU (+5.6%), MathVista (+17.0%) and DocVQA (+7.7%), although MolmoE-1B is specialist model optimized for image understanding. At the 3B scale, SF-LLaVA-115-3B also demonstrates superior results, (e.g,, outperforming 2hii3..-Vision--2- by +1.7% on MMMU, +1.77% on Math and +1.0% TextVQA).
1B outperforms Intern
VI.2.5-1B InternV1.2.551
Second, SF-LLa outperforms strong baselines the 7B scale, except for Intern VL2.5 and Qwen2-VL. Using MM1.5-70 an example,5F-LLLaA-11. achieves better results across benchmarks (e.g., +7.2% on MMMU, +12.5% on MM-Vet, and and impressed by the superior results of Intern and Qwen2-VL, especially on TextRich. We hypothesize it is due to our (i) lower input resolution (e.g., 2.36M vs. 9.63M of VL2.5)) (ii) fewer training stages (egg,, , 2 3 Qwen2-VL) and (iii) frozen vision encoder. This aligns with prior findings (Zhang et al., 2025b) that, when the model size gets larger, higher input resolution and more training stages with fully tunable parameters are pivotal for improving the image performance. Given that our model is video-centric and these enhancements significantly increase training costs, we leave their exploration for future work.
SF-LLaVA-115
+ +12.9% on OCRBench). We are
Third, SF-LLaVA-1.5's image capability benefits from joint video-image training. SF- LLaVA-1.5, jointly optimized on video and image data, outperforms ST-LLaVA--.55lmage on most benchmarks. To confirm the improvements are not solely due to longer training, we conduct second-stage training for SF-LLaVA-1.5-mmaa the performance gap remains, indicating that joint training is the primary factor. Addi- tionally, the improvements are more significant on Knowledge and General benchmarks (e.g,, +1.2% MMMU and +10.1% % MM-Vet at the 1B scale). We hypothesize this is
using only image data. However,
7

---

effectiveness in balancing strong Published as a conference paper at COLM 2025
Video-MME (w/o sub)
LongVideoBeechh (val)
Structure
(short)
(med)
(long)
(avg)
Group-based SlowFast (GSF)
64.4
52.8
46.1
54.4
52.7
Interleaved FlowFast (ISF)
52.3
Table 4: Comparison between GSF and ISF on video understanding.
Slow Frames Na
Fast Frames N
Total Frames
Input Token
Video-MME (w/o sw/o.uu))
sub) (long)
LongVideollench (val)
N
(short)
(med)
(avg)
32
0
32
7K
62.0
50.4
44.1
32.1
52.4
0
10K
64,9
51.1
45.0
33.7
52.5
64
0
64
14K
64.3
51.0
45.5
53.6
52.2
128
63.0
53.3
0
128
128
2K
59.3
49.7
44.3
51.1
49.7
128
128
64.4
52.8
46.1
34.4
52.7
Table 5: Results of SF-LLaVA-1.5 with different design choices on video understanding.
because our video data mainly comes from lifestyle scenarios, which could not directly benefit text-rich tasks. A deeper analysis of joint training will be provided Sec. 4.4.2.
4.4
Ablation Studies
All ablation studies are conducted on the 1B model with our default settings (Sec. 4.1). To save training costs, models are trained on 1.2M image and 600K video samples, randomly selected from our original data mixture (Appendix A.1). The performance is evaluated on Video-MME and LongVideo VidcoBench to cover both short and long videos.
4.4.1 Design Choices of SlowFast
Group-based FlowFast (GSF) vs. Interleaved SlowFast (ISF). We introduced these Slow structures in Sec. 3.1.1 and report their video understanding results in Table 4. GSF and ISF perform comparably on Video-MME (54.4% vs. 54.1% on average) and Bong (52.7% vs. 52.3%), suggesting that SF-LLaVA-1.5 is not sensitive to this design choice. This highlights the general effectiveness of the SlowFast approach in improving long-form video understanding. Since GSF consistently achieves superior performance across benchmarks, we adopt as the default SlowFast structure in this paper.
Effect of the Slow and Fast Pathways. First, we assess the necessity of the Slow and Fast pathways by removing them individually. Table shows that SF-LLaVA-115 outperforms both Slow-only (row vs. row 6) and Fast-only (row 5 vs. row 6) models. This is expected since they use fewer input frames or tokens than the full model. Second, we test if Slow remains more effective when the Slow-only model uses a comparable number of input tokens (e.g., 48 frames with ~00K tokens). The results (row 2 vs. row 6) demonstrate that SlowFast outperforms this baseline (e-g., +1.1% on Video-MME long), indicating that the improvements are not merely due to using more information. Third, we argue that SlowFast enhances both computational efficiency and long-range range temporal modeling. We verify this by comparing SlowFast with the Slow-only model that uses the same number of input frames (e.g,, Nslow = N = 128). The results (row 4 vs. row 6) show that SlowFast maintains superior performance while using only ~30% (9K vs. 28K) of its input tokens.
Flow vs. Other Video Projectors. We compare SlowFast with existing video projectors in Table 6. Specifically, we apply 2 average pooling in Spatial Pooling and Dynamic Compressor and follow Apollo (Zohar et al., 2024) by using 16 tokens per frame in Q-Former and Perceiver Resampler. All models process up to 128 input frames. Compared Pooling and Dynamic Compressor, FlowFast improves runtime by 25% while surpassing them across all benchmarks. also significantly outperforms Q-Former and Perceiver Resampler, which use fixed-length tokens for information compression, limiting their ability to handle long video sequences. Moreover, Q-Former and Perceiver Resampler introduce additional parameters (e.g., BERT-Base in Q-Former), which restrict their advantage in runtime efficiency, These results demonstrate video performance and computational efficiency.
Spatial
8

---

Published as a conference paper at COLM 2025
Input Token #
Runtime (per video)
Video-MME sub) (long)
Long LongVideollench (val)
Video Projector
(short)
(med)
(avg)
Spatial Puoling (Xu et al. 2024a)
28K
2.40s
63.3
51.8
45.7
53.6
51.7
Dynamic Compressor (1.iu al., 2025)
2.45s
63.5
52.4
45.8
53.9
Qformer (Li et al, 2023a)
2K
1.59s
46.7
43.0
38.4
42.7
45.0
Perceiver Resampler (Jaegle et al., 2021)
2K
1.50s
52.8
45.9
43.0
472
48.4
SlowFast
9K
1.79s
64.4
52.8
46.1
50.4
52.7
Table 6: Comparison between SlowFast and existing video projectors on video under- standing. All models take 128 frames as inputs. The runtime (per video) measures only the model's is forward pass on single H100-80G GPU, using the Vongeideeeench dataset.
Video Benchmarks
Image Benchmarks
Ratio
Video-MME (w/o sub) (long)
BongVideoBench (val)
MMMU (val)
OCRBench (test)
TextVQA
RW-QA
(short)
(med)
(avg)
(val)
0:2
60.6
47.3
$1.6
49.8
48.2
38.4
58.4
66.9
69.4
1:00
63.4
44.3
52.0
39.4
2:0
63.7
52.0
44.9
53.5
52.4
39.3
56.6
61.9
64.6
1:5
65.1
50.1
45.9
53.7
52.3
44.0
59.0
66.2
69.2
1:1
64.8
30.5
45.3
53.5
52.1
39.9
58.5
68.3
69.5
1:2
64.4
52.8
46.1
54.4
52.7
40.0
59.1
68.2
69.7
1:3
63.7
32.3
54.0
52.5
58.8
68.0
Table 7: Results of using different video-to-image data ratios in joint training.
4.4.2 Design Choices of Model Training
Effect of Video-to-Image Ratio in Joint Training. We examine the optimal video-to-image ratio by fixing video samples at 600K and evaluating the impact of varying image samples. Specifically, we explore the following ratios {0:2, 1:0, 2:0, 1:0.5, 1:1, 1:2, 1:3), where a ratio of "1:0" and "2:0" use only video data. Results are shown in Table with the following findings. First, training with only video data clearly decreases the performance in image understanding (row 2 vs. row 4), with substantial drop on text-rich benchmarks (e.g., -5.0% on TextVQA). Second, joint video-image training generally improves SF-LLaVA-1.5's video capability (row 2 vs. row 6), such as on Video-MME (53.2% 2% vs. 54.4% on average). Third, increasing the proportion of image data does not always lead to better video results (row row 7). Fourth, video-to-image ratio of "1:2" achieves the best overall performance in video and image understanding, which we adopt in our final data mixture.
5 Limitations
First, SF-LLa prefers FPS sampling, but falls back to uniform sampling when the video duration exceeds the maximum frame capacity (i.e., 128 in this paper). This approach may miss some key frames in long-form videos and mislead the model about a video's 's playback speed (e.g,, A ten-minute video and one-hour video have the same number of input frames). Developing an efficient memory model to summarize the long-range context is promising direction (Xu et al., 2021). We can also input extra information (egg., frame timestamps) to enhance the temporal modeling. Second, SF-LLaVA-1.5's performance can be further improved by tuning all parameters, including the visual encoder. However, we found this is not trivial for Long Video LLMs due to the high GPU memory cost of caching the activation values. Future studies could explore the integration of memory-savine techniques, such as Stochastic BP (Cheng et al., 2022). More analysis will be discussed in Appendix A.4.
Conclusion
Building upon the insights of SlowFast-LLava (Xu al., 2024b), we introduce SlowFast- LLaVA-1.55, a new family of token-efficient Video LLMs for long-form video understanding. While SlowFast-LLLaV adapts the two-stream SlowFast inputs into training-free model, this work explores further improvements by building a supervised fine-tuning pipeline with a high-quality data mixture. Our model family, ranging from 1B to 7B parameters, focuses on developing lightweight models that are both compact for potential edge deployment and powerful for various video tasks. Experimental results demonstrate that Slow Fast-LLaVA- 1.5 achieves superior performance across video benchmarks while maintaining strong image capabilities. We hope our work inspires the community to develop efficient yet robust Long Video LLMs based open-source datasets.

---

Published as a conference paper at COLM 2025
Acknowledgments
We thank Yizhe Zhang, Feng Tang, Jesse Allardice, Jiaming Hu, Yihao Qian, Zhe Fu, Hong- You Chen, Wentao Wu, Junting Pan, Bowen Zhang, Yanghao Li for their kind help.
References
https://haggingfacccco
htt.:/tt////mmmms//////rroororor printing.
wendlerc/renderrctex..
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadal- lah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. ar arXiv:2404.42211 2024.
Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. In AAAI, 2019.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Tlge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. GPT-4 technical report. arXiv:2303.08774, 2023.
Gilles Baechler, Srinivas Sunkara, Maria Wang, Fedir Zubach, Hassan Mansoor, Vincent Etter, Victor Cärbune, Jason Lin, Jindong Chen, and Abhanshu Sharma. Screenai: A vision-language model for ui and infographics understanding. arXio:2402.04615, 2024.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. 13X1v:2502.13923, 2025.
Jonas Belouadi, Simone Paolo Ponzetto, and Steffen Eger. DeTikZify: Synthesizing graphics programs for scientific figures and sketches with TikZ. In NeurlPS, 2024.
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In ICCV, 2019.
Jie Cao and Jing Xiao. An augmented benchmark dataset for geometric question answering through dual parallel text encoding. In COLING, 2022.
Junbum Cha, Wooyoung Kang, Jonghwan Mun, and Byungseok Roh. Honeybee: Locality- enhanced projector for multimodal Ilm. In CVPR, 2024.
Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler- Lussier, and Ningchuan Xiao. Mapqa:
A dataset for question answering choropleth maps. In NeurIPS Workshop, 2022.
Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. Unigeo: Unifying geometry logical reasoning via reformulating mathematical expression. arXin:2212.02746, 2022.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. SharegptAv: Improving large multi-modal models with better captions. In ECCV, 2024a.
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. Sharegpt4video: Improving video understanding and generation with better captions. wwXio:2406.04325 2024b.
Shimin Chen, Xiaohan Lan, Yitian Yuan, Zequn Jie, and Lin Ma. Timemarker: A versatile video-Ilm for long and short video understanding with superior temporal localization ability. arXio:2411.18211 2024c.
10

---

Published as a conference paper at COLM 2025
Wenhu Chen, Hongmin Wang, Jianshu Chen, Yunkai Zhang, Hong Wang, Shiyang Li, Xiyou Zhou, and William Yang Wang. Tabfact large-scale dataset for table-based fact verification. In ICLR, 2020.
Xinlei Chen, Hao Fang, Isung-Y Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. ar XrXi::1504.00325, 2015.
Yukang Chen, Fuzhao Xue, Dacheng Li, Qinghao Hu, Ligeng Zhu, Xiuyu Li, Yunhao Fang, Haotian Tang, Shang Yang, Zhijian Liu, et al. Longvila: Scaling long-context visual language models for long videos. ar arXiv:2408.1013:
2024d.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling, arXiv:2412.05271, 2024e.
Zhe Chen, Weiyun Wang, Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. ar arxiv:2004. 16821, 2024f.
Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan Routledge, et al. Finqa: A dataset of numerical reasoning over financial data. arXio:2109.0012) 2021.
Feng Cheng and Gedas Bertasius. Tallformer: Temporal action localization with long- memory transformer. In ECCV, 2022.
Feng Cheng, Mingze Xu, Yuanjun Xiong, Hao Chen, Xinyu Li, Wei Li, and Wei Xia. Stochastic backpropagatton: A memory efficient strategy for training video models. In CVPR, 2022.
Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. VideoLLaMA 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms. 247:27406.77776,, 2024.
Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian- Guang Lou, and Dongmei Zhang. Hitab: A hierarchical table dataset for question answering and natural language generation. arXiv:2108.067 2021.
Erfei Cui, Yinan He, Zheng Ma, Zhe Chen, Hao Tian, Weiyun Wang, Kunchang Li, Yi Wang, Wenhai Wang, Xizhou Zhu, Lewei Lu, Tong Lu, Yali Wang, Limin Wang, Yu Qiao, and Jifeng Dai. Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2024. URL https://sharegpt4o. gi thub.
Brian Davis, Bryan Morse, Scott Cohen, Brian Price, and Chris Tensmeyer. Deep visual template-free form parsing. In ICDAR, 2019.
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mo- hammad Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models. 24X9::2409 2024.
Zane Durante, Qiuyuan Huang, Naoki Wake, Ran Gong, Jae Sung Park, Bidipta Sarkar, Rohan Taori, Yusuke Noda, Demetri Terzopoulos, Yejin Choi, et al. Agent Al: Surveying the horizons of multimodal interaction. arXio:2401.03568, 2024.
Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, and Hui Wang. Video-ccam: Enhancing video-language understanding with causal ross-attention masks for short and long videos. arXi::2408.1 14023, 2024.
11

---

Published as a conference paper at COLM 2025
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever compre- hensive evaluation benchmark of multi-modal Iims in video analysis. arXio:2405.2107, 2024.
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang, Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, et al. G-llava: Solving geometric problem with multi-modal large language model. ar arXio:2312.11370, 2023.
Lishuai Gao, Yujie Zhong, Yingsen Zeng, Haoxian Tan, Dengjie Li, and Zheng Zhao. Linvt: Empower your image-level large language model to understand videos. ar arXiv:2412.05185, 2024.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, 2018.
Rujun Han, Peng Qi, Yuhao Zhang, Lan Liu, Juliette Burger, William Yang Wang, Zhiheng Huang, Bing Xiang, and Dan Roth. Robustqa: Benchmarking the robustness of domain adaptation for open-domain question answering. In ACL Findings, 2023.
Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar. lcdar2019 competition on scanned receipt ocr and information extraction. In ICDAR, 2019.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. ICML, 2021.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
Kushal Kafle, Scott Cohen, Brian Price, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In CVPR, 2018.
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning. ar arXiv:1710.07300, 2017.
Mehran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Sori- cut. Geomverse: A systematic evaluation of large models for geometric reasoning. arXiv:2312.12221, 2023.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In ECCV, 2016.
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than sixth grader? textbook question answering for multimodal machine comprehension. CVPR, 2017.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. NeurlPS, 2020.
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeong Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In ECCV, 2022.
Wonkyun Kim, Changin Choi, Wonseok Lee, and Wonjong Rhee. An image grid can be worth video: Zero-shot video question answering using a vlm. arXiv:2403. 18406,, 2024.
Jason Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 2018.
12

---

Published as a conference paper at COLM 2025
Hugo Laurençon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? (iX::22455.02246, 2024a.
Hugo Laurençon, Léo Tronchon, and Victor Sanh. Unlocking the conversion of web screen- shots into html code with the websight dataset. ar -XXi::2433000029 2024b.
Hosu Lee, Junho Kim, Hyunjun Kim, and Yong Man Ro. Look every frame all at once: Video-ma2mba for efficient long-form video understanding with multi-axis gradient checkpointing. 1ttt::2111.146000 2024a.
Seon-Ho Lee, Jue Wang, Zhikang Zhang, David Fan, and Xinyu Li. Video token merging for long-form video understanding, ar #rXiv:2410.23782, 2024b.
Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Ilava-onevision: Easy visual task transfer. 1xXi:2408.03326, 2024a.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In ICML, 2023a.
Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. VideoChat: Chat-centric video understanding. arXie:2305.06355, 2023b.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, Limin Wang, and Yu Qiao. MVBench: A comprehensive multi-modal video understanding benchmark. arXit:2311.17005 2023c
Lei Li, Yuqi Wang, Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision- language models. ar drXii:2403.00231, 2024b.
Qingyun Li, Zhe Chen, Weiyun Wang, Wenhai Wang, Shenglong Ye, Zhenjiang Jin, et al. Omnicorpus: A unified multimodal corpus of 10 billion-level images interleaved with text. XrX:e:2406.088 084488, 2024c.
Xinhao Li, Yi Wang, Jiashuo Yu, Xiangyu Zeng, Yuhan Zhu, Haian Huang, Jianfei Gao, Kunchang Li, Yinan He, Chenting Wang, et al. Videochat-flashi Hierarchical compression for long-context video modeling. arXii:2501.00574 2024d.
Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamín Van Durme, and Alan L Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning. In CVPR, 2023d.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-LLaVA: Learning united visual representation by alignment before projection. arXio:2311.10122 2023a.
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. arXiv:2312.07533 2023b.
Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, and Maosong Sun. Streamingbench: Assessing the gap for milms to achieve streaming video understanding. arXiv:2411.0366 2024.
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. wxXix:2311.0757 2023c.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. ACL, 2023a.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. 1rXt::2306.14565, 2023b.
13

---

Published as a conference paper at COLM 2025
Fuxiao Liu, Xiaoyang Wang, Wenlin Yao, Jianshu Chen, Kaiqiang Song, Sangwoo Cho, Yaser Yacoob, and Dong Yu. Mmc: Advancing multimodal chart understanding with large-scale instruction tuning. xfXio:2311.1077, 2023c.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. ar arXf::2310.03744, 2023d.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In NeurlPS, 2023e.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. LLaVA-NeXT: Improved reasoning, oct, and world knowledge, 2024a. URL https: //llava-vl.githhb. http://ww.ggiuubb..oo/o///////// -1--4-01---011111 30-11ava-next/.
Jiajun Liu, Yibing Wang, Hanghang Ma, Xiaoping Wu, Xiaoqi Ma, Xiaoming Wei, Jianbin Jiao, Enhua Wu, and Jie Hu. Kangaroo: A powerful video-language model supporting long-context video input. (XXi::2408115442 2024b.
Yuanxin Liu, Shicheng Li, Yi Liu, Yuxiang Wang, Shuhuai Ren, Lei Li, Sishuo Chen, Xu Sun, and Lu Hou. Tempcompass: Do video llms really understand videos? arXio:2403.00476, 2024c.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 2024d.
Zhijian Liu, Ligeng Zhu, Baifeng Shi, Zhuoyang Zhang, Yuming Lou, Shang Yang, Haocheng Xi, Shiyi Cao, Yuxian Gu, Dacheng Li, et al. Nvila: Efficient frontier visual language models. 7xXii:2412.04466 2024e.
Zuyan Liu, Yuhao Dong, Ziwei Liu, Winston Hu, Jiwen Lu, and Yongming Rao. Oryx MLLM: On-demand spatial-temporal understanding at arbitrary resolution. ICLR, 2025.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song- Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. In ACL, 2021a.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In NeurIPS Track on Datasets and Benchmarks, 2021b.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS, 2022a.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXio:2209.14660, 2022b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao
Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathe- matical reasoning of foundation models in visual contexts. In ICLR, 2024.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. VideoGPT+: Inte- grating image and video encoders for enhanced video understanding. arXio:2406.09418, 2024a.
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. Video- ChatGPT: Towards detailed video understanding via large vision and language models. In ACL, 2024b.
Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro von Werra, and Thomas Wolf. Smolvim: Redefining small and efficient multimodal models. 2025.
14

---

Published as a conference paper at COLM 2025
U-V Marti and Horst Bunke.
The iam-database: an english sentence database for offline handwriting recognition. IJDAR, 2002.
Ahmed Masry, Do Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. ChartQA: A benchmark for question answering about charts with visual and logical reasoning. In ACL Findings, 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In CVPR, 2021.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. In WACV, 2022.
Brandon McKinzie, Zhe Gan, Jean-Phillippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. MM1: Methods, analysis & insights from multimodal Ilm pre-training. arXio:2403.06611, 2024.
Nitesh Methani, Pritha Ganguly, Mitesh M. Khapra, and Pratyush Kumar. Plotqa: Reasoning over scientific plots. In WACV, 2020.
Anand Mishra, Karteek Alahari, and CV Jawahar. Scene text recognition using higher order language priors. In BMVC, 2012.
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty, Ocr-vqa: Visual question answering by reading text in images. In ICDAR, 2019.
Harold Mouchere, Christian Viard-Gaudin, Dae Hwan Kim, Jín Hyung Kim, and Utpal Garain. Crohme2011: Competition on recognition of online handwritten mathematical expressions. In ICDAR, 2011.
Jason Obeid and Enamul Hoque. Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model. xfX1::201000911222222 2020.
OpenAl. Gpt-4v, 2023. URL http:://.gppaaaa.c////////////es
OpenAl. Gpt-4o, 2024. URL http:://openas.ccooo/nnne///lllp
Panupong Pasupat and Percy Liang. Compositional semantic parsing on semi-structured tables. arXi::1508.0030 2015.
Renjie Pi, Jianshu Zhang, Jipeng Zhang, Rui Pan, Zhekai Chen, and Tong Zhang. Im- age textualization: An automatic framework for creating accurate and detailed image descriptions. arXio:2406.07502 2024.
Viorica Păträucean, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Miech, Alex Frechette, Hanna Klimezak, Raphael Koster, Junlin Zhang, Stephanie Winkler, Yusuf Aytar, Si- mon Osindero, Dima Damen, Andrew Zisserman, and João Carreira. Perception test: A diagnostic benchmark for multimodal video models. In NeurIPS, 2023. URL https://openrevias
met/forum?id-HHEGKKK
Rui Qian, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Shuangrui Ding, Dahua Lin, and Jiaqi Wang. Streaming long video understanding with large language models. NeurIPS, 2024.
Ruchit Rawal, Khalid Saifullah, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. Cinepile: A long video question answering dataset and benchmark. arXiv:2405.0881. 2024.
Michael S Ryoo, Honglu Zhou, Shrikant Kendre, Can Qin, Le Xue, Manli Shu, Silvio Savarese, Ran Xu, Caiming Xiong, and Juan Carlos Niebles. xgen (blip-3-video): You only need 32 tokens to represent video even in vlms. arXin:2410.16267, 2024.
Minjoon Seo, Hannaneh Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. Solving geometry problems: Combining text and diagram interpretation. In EMNLP, 2015.
15

---

Published as a conference paper at COLM 2025
Xiaoqian Shen, Yunyang Xiong, Changsheng Zhao, Lemeng Wu, Jun Chen, Chenchen Zhu, Zechun Liu, Fanyi Xiao, Balakrishnan Varadarajan, Florian Bordes, et al. Longvu: Spatiotemporal adaptive compression for long video-language understand- ing. arXit:2410.17434, 2024.
Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, Humphrey Shi, et al. Eagle: Exploring the design space for multimodal lims with mixture of encoders. ar (rXio:2408.15999, 2024.
Chenglei Si, Yanzhe Zhang, Zhengyuan Yang, Ruibo Liu, and Diyi Yang. Design2code: How far are we from automating front-end engineering? orXiv:2403.0316, 2024.
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioningwith reading comprehension. In ECCV, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In CVPR, 2019.
Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, and Tal Hassner. TextOCR: Towards large-scale end-to-end reasoning for arbitray-shaped scene text. In CVPR, 2021.
Enxin Song, Wenhao Chai, Guanhong Wang, Yucheng Zhang, Haoyang Zhou, Feiyang Wu, Haozhe Chi, Xun Guo, Tian Ye, Yanting Zhang, al. Moviechat: From dense token to sparse memory for long video understanding. In CVPR, 2024.
Stanislwwa,,, Filip Graliñski, Anna Wróblewska, Dawid Lipiński, Agnieszka Kaliska, Paulina Rosalska, Bartosz Topolski, and Przemystaw Biecek. Kleister: key information extraction datasets involving long documents with complex layouts. In ICDAR, 2021.
Stacey Svetlichnaya. Deepform: Understand structured documents at scale-wandb. ai, 2020.
Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmre: Machine reading comprehen- sion on document images. In AAAI, 2021.
Benny J. Tang, Angie Boggust, and Arvind Satyanarayan. VisText: A Benchmark for Semantically Rich Chart Captioning. In ACL, 2023.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXio:2312.11805, 2023.
Peter Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Adithya Jairam Vedagiri IYER, Sai Charitha Akula, Shusheng Yang, Jihan Yang, Manoj Middepogu, Ziteng Wang, et al. Cambrian-1: fully open, vision-centric exploration of multimodal 1lms. NeurlPS, 2025.
Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, and Saining Xie. Eyes wide shut? exploring the visual shortcomings of multimodal 1lms. In CVPR, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. ar xrXio:2302.13971, 2023.
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li. Screen2words: Automatic mobile ui summarization with multimodal learning. In UIST, 2021.
Haibo Wang, Bo Feng, Zhengfeng Lai, Mingze Xu, Shiyu Li, Weifeng Ge, Afshin Dehghan, Meng Cao, and Ping Huang. Streambridge: Turning your offline video large language model into a proactive streaming assistant. arXio:2505.05467 2025.
Jiawei Wang, Liping Yuan, Yuchen Zhang, and Haomiao Sun. Tarsier: Recipes for training and evaluating large video description models. arXim:2407.006 00644, 2024a.
16

---

Published as a conference paper at COLM 2025
Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see is to believe: Prompting gpt-4y for better visual instruction tuning. ar arXiv:331.00777, 2023.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. ar (xXi::2499.12199, 2024b.
Weihan Wang, Zehai He, Wenyi Hong, Yean Cheng, Xiaohan Zhang, Qi, Xiaotao Gu, Shiyu Huang, Bin Xu, Yuxiao Dong, et al. Lvbench: An extreme long video understanding benchmark. ar WXXr:2406.08035, 2024c.
Haoning Wu, Dongxu Li, Bei Chen, and Junnan Li. Longvideobench: A benchmark for long-context interleaved video-language understanding. NeurlPS, 2025.
Wenhao Wu. VA: Offline milm training-free video assistant. arXfo:2405.07799 2024.
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. NEXT-QA: Next phase of question-answering to explaining temporal actions. In CVPR,
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijie Lin, See Kiong Ng, and Jiashi Feng. PLLaVA: Parameter-free llava extension from images to videos for video dense captioning. ar 7Xit:2404.16994) 2024a.
Mingze Xu, Yuanjun Xiong, Hao Chen, Xinyu Li, Wei Xia, Zhuowen Tu, and Stefano Soatto. Long short-term transformer for online action detection. NeurIPS, 2021.
Mingze Xu, Mingfei Gao, Zhe Gan, Hong- You Chen, Zhengfeng Lai, Haiming Gang, Kai Kang, and Afshin Dehghan. SlowFast-LLLaA, A strong training-frit ffree baseline for video large language models. arXiv:2407.15847 2024b.
You
Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang. Vision-flan: Scaling human-labeled tasks in visual instruction tuning. arXie:2402.11690 2024c.
Le Xue, Manli Shu, Anas Awadalla, Jun Wang, An Yan, Senthil Purushwalkam, Honglu Zhou, Viraj Prabhu, Yutong Dai, Michael S Ryoo, et al. xgen-mm (blip-3): A family of open large multimodal models. ar wxXx:2408.08877 2024.
Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, et al. Magma: A foundation model for multimodal ai agents. arXiv:2502.1313.) 13130, 2025.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Jang, Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: gpt-4v level mllm on your phone. arXiv:2408. 01800, 2024.
Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfeng Tian, Qi Qian, Ji Zhang, et al. Ureader: Universal ocr-free visually-situated language understanding with multimodal large language model. arXtoo2310.05116, 2023.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity, ar xxX0:2230.00704, 2023.
Keen You, Haotian Zhang, Eldon Schoop, Floris Weers, Amanda Swearngin, Jeffrey Nichols, Yinfei Yang, and Zhe Gan. Ferret-Ul: Grounded mobile ui understanding with multimodal lims. In ECCV, 2024.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. In JCML, 2024.
17

---

Published as a conference paper at COLM 2025
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. ActivityNet-QA: dataset for understanding complex web videos via question answering. In AAAI, 2019.
Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji, Zhongqin Wu, and Xi- ang Bai. Syntax-aware network for handwritten mathematical expression recognition. ar aXXiv:2203.01601, 2022.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In ICCV, 2023.
Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. VideoLLaMM 3: Frontier multi- modal foundation models for image and video understanding, ar arXiv:2501.13106, 2025a.
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chur Zhu. Raven: A dataset for relational and analogical visual reasoning. In CVPR, 2019.
Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuueed audio-visual modell language model for video understanding. ar wwx:::2300600888, 2023.
Haoji Zhang, Yiqin Wang, Yansong Tang, Yong Liu, Jiashi Feng, Jifeng Dai, and Xiaojie Jin. Flash-vstream: Memory-based real-time understanding for long video streams. ar xrXiv22406.08055 2024a.
Haotian Zhang, Mingfei Gao, Zhe Gan, Philipp Dufter, Nina Wenzel, Forrest Huang, Dhruti Shah, Xianzhi Du, Bowen Zhang, Yanghao Li, et al. MM1. 5: Methods, analysis insights from multimodal Ilm fine-tuning, ICLR, 2025b.
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. Internlm (remmimmxxoomoser-22.. A versatile large vision language model supporting long-con ong-contextual input and output. arXiv:2407.03320 2024b.
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. Xiix:2406.16852, 2024c.
Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, Peng Gao, and Hongsheng Li. Mavis: Mathematical visual instruction tuning. In arx::22407.088 008399 2024d.
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward. arXiv:2404.01255, 2024e.
Shi-Xue Zhang, Xiaobin Zhu, Lei Chen, Jie-Bo Hou, and Xu-Cheng Yin. Arbitrary shape text detection via segmentation with probability maps. TPAMI, 2022.
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. LLaVA-NeXT: A strong zero-shot video understanding model, 2024f. URL https::///// //11://-/laaaaaaa git. 1001h///00000///0//000000--0-0-0 10avvaa--e newtwa-nxxvvieeo/....
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruction tuning with synthetic data. arXio:2410.02211, 2024g.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. 10X0011000.00103, 2017.
18

---

Published as a conference paper at COLM 2025
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-tase long video understanding. ar 4rX::22406.0426, 2024.
Fengbin Zhu, Wenqiang Lei, Youcheng Huang, Chao Wang, Shuo Zhang, Jiancheng Lv, Fuli Feng, and Tat-Seng Chua. Tat-qa: A question answering benchmark on a hybrid of tabular and textual content in finance. arXiv:2105.07624, 2021.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In CVPR, 2016.
Orr Zohar, Xiaohan Wang, Yann Dubois, Nikhil Mehta, Tong Xiao, Philippe Hansen-Estruch Licheng Yu, Xiaofang Wang, Felix Juefei-Xu, Ning Zhang, et al. Apollo: An exploration of video understanding in large multimodal models. arXin:2412.10360, 2024.
19

---

Published as a conference paper at COLM 2025
A
Appendix
A.1 Details of Data Mixture
Mixture
Data Category
Datasets
Samples
LLVVA Complex Reasening (Luu et al., 2023e),
LLaVA Conversation (Liu 2023e),
ShareGiPI-lv (Chen et al, 2024aa,, Coco Caption (Chen et 2015),
LLaVA v1. VQA2 OKVQA (Liu ct 20233))
LLaVA v15 GQA
2023d),
LLaVA v1.5 A OKVQA (Liu et 2023d),
General
Memmo-Ask-MMdel-Annnee
(Deilke et al, 2024),
Image Textualization
2024), do
al. 2024),
Vision FLAN (Xu et al., 2024c), VizWiz (Gurari 2018).
TallyQA (Acharya et 2019), Visual7W (Zhu et 2016),
VQARAD (Lau ct. 2018), VSR (Liu
2123a),
Hateful Memes (Kiela et al. 2020)
OCRVQA (Mishra
2019), Synthdog- En (Kim al., 2022),
TextCaps (Sidorov et 2020),
(Singh an 2019),
DVQA (Kafle el al. 2018), CharlQA (Masry al,, 2022),
DocVQA (Mathew et al., 2021), InfoVQA (Mathew et al. 2022),
VisuaMMRC (Tanaka
2021), WikiTQ (fasuupat & Linng. 2015).
DeepForm (Svellichnaya, 2020), KleislerCharity (Slanistawek al., 2021),
TabFact (Chen al., 2020), SA (Baechler et al, 2024),
Tab (Lu al,, 2022b), ST-VQA (Biten el 2019).
VisText (Tang et al. 2023), HiTab (Cheng et al., 2021).
AndiOA (li al., 2024bb), WikisQL (Zhong ct 2017),
Image Mixture
TextRich
Chart2Texl (Obeid & Hoque, 2020), RenderedText (ren),
4.67M
FinQA (Chen et al. 2021). TAT-QA (Zhu et al, 2022),
Pixmo-Does (Delike al,
PloIQA (Methani
2020),
MMC-Instruct (Liu et al., 2023c), ArI (Zhang et al., 2022),
NAF (Davis et al., 201 9), SROIE (Huang et al,, 2019).
LRV Chart (Liu el al., 20236), FigureQA (Kahou el al,, 2017),
RoBUT SQA (Han al., 2023), Screen2Words (Wang al., 2021),
HME100K (Yuan et al., 2022). UReader (Ye et 2023).
Diagram ImageZText (Laurençon
2024a), ChromeWriting (Mouchere et 2011),
IIITSK (Mishra et al., 2012), 1AM (Marti Bunke, 2002),
TextOCR (Singh et al., 2021), K12 Printing (k12)
(Kembhavi
2016), ScienceQA
al., 2022a).
Geom GeomVerse Verse (Kazemi et al,, 2023), CT EVER (Johnson al,, 2017),
IconQA (Lu et al., 2021b), RAVEN (Zhang al, 2019),
Inter-GPS
al., 2021a), WebSight (Laurenion
2024b),
DaTikZ (Belouadi al., 2024), Design2Code (Si et al., 2024),
Kowledge
TQA (Kembhavi al., 2017), MAVI5 MCollect (Zhang al., 2024d; Li al., 2024a), Geol
2024a),
MAVIS Engine (Zhang al., 2024d;
(Gao 2023),
Geo170K Align (Gao el al., 2023; Li el al., 2024a), GeometryaK (Lu el al,, 2021a),
GEOS (Seo al. 2015), GeoQA+ (Cao & Xiso, 2022).
MapQA (Chang ot 2022), Super CLEVR
al., 2023d),
UniGeo (Chen et al,, 2022)
LLVVA-Hound (Zhang al,, 2024e), ShareGPT4Video (Chen 2024b),
VideoChatGPT-PPe (Mazz 2024a), LLaVA-Video-172 (Zhang al., 2024g),
Video Mixture
General
Cinepile (Rawal et al., 2024), NetivityNet-Ok AA (Yu et 2019),
2.01M
NEXT-QA (Niso 2021), Perception Test (Patnáucean 2023)
Table 8: Details of our image and video mixtures.
A.2 Benchmarks and Metrics
All evaluations are performed using the 1mms-eval3 toolkit, where we use the official evaluation metrics to report numbers without any filtering on the prediction outputs.
Category
Benchmark
# Videos
#QAs
Arg Duration (s)
Video-MME (Fu al., 2024)
900
2700
1010
Perception Test (val) (Patráucean et al,, 2023)
5900
19139
23
General Video QA
ActividyNNe--AA (test) eY 2019)
8000
180
VCGBench (test) Mauz et al. (202tb)
800
3497
180
Long VideoBench (val) (Wu al, 2025)
752
1337
473
Long-Form Video Understanding
MLVU (tes) (Zhou al., 2024)
1730
3102
930
L.Bennch (test) (Wang et al., 2024c)
103
1549
4101
TempCompass (me) et al., 2024c)
7540
Temporal Remsning
NExT-QA (mc) (Xiao et al., 2021)
1000
8564
44
Table 9: Details of video understanding benchmarks.
htttss//ggthhb.cco///oooongg/// lall--evvll
20

---

Published as a conference paper at COLM 2025
A.2.1 Video Benchmarks
We evaluate our model on various video understanding benchmarks in Table 9.
A.2.2 Image Benchmarks
We evaluate our model on the following image understanding benchmarks:
Knowledge Image QA inspects model's capability of answering questions requiring knowledge in specific domains. Our model is evaluated on A12D (Kembhavi et al., 2016) and ScienceQA (Lu al., 2022a) for science, MathVISTA Lu et al. (2024) for math and MMMU (Yue et al., 2024) for multi-discipline tasks.
General Image QA evaluates the general image capability of our model. We select RealWorldQA and MMVet (Yu et al., 2024) to serve this purpose, where RealWorld examines model's capability in real-world scenarios and MMVet assesses model's performance for more complicated tasks.
TextRich Image QA contains images embeded with dense texts. To achieve high performance, a model is expected to excel reasoning over reading. We include OCRBench (Liu et al., 2024d), TextVQA (Singh et al., 2019) and DocVQA (Mathew et al., 2021) measuring OCR, scene text and document understanding, respectively.
A.2.3 Instantiations of SlowFast Cont'd
Group-based
(GSF)
Sampled Slow Frames
Sampled Fast Fremes
Slow pathway tokens
Interleaved SlowFast (ISF)
pathway
6
6
Learnable special token
Sampled Video Frames
Figure Visualization of Group-based SlowFast (GSF) and Interleaved SlowFast (ISF).
A.3 More Video Understanding Results
We compare with recent Video LLMs as representative examples Table 2, and here, we in- clude broader group of models in Table 10. For ActivityNet-QA and VCGBench, we adopt the GPT-assisted evaluation to assess the accuracy. Specifically, we use UPT-3.55Turee as the judge. It is worth noting that model cannot be directly compared with previous work that uses GPT-3.5-Turbo S-Turbo-0613 (deprecated by OpenAl) or an unknown version, since different GPT versions significantly impact the results (Wu, 2024).
A.4 Effect Training the Visual Encoder
By default, the visual encoder is frozen in both Stage and II. We now assess whether training the visual encoder improves the image and video understanding performance.
We start with training Stage I, tuning the visual encoder together with other parameters (named as Vo-LLaVA-15--maae--BB presented in Table 11. We observe that training the visual encoder significantly improves
We evaluate it image benchmarks, with results
https///hugginfcccooodddaeeee Realworld/Raa
21

---

Published as a conference paper at COLM 2025
General VideoQQA
Long-Form Video
Understanding
Reasoning
Jemporal
Long LongMdeniBeachh
ActivtyNer-OO
VideoMMMt
VideoMMit
Pecoop
VCGRend
LVBeench
TompCamp
NRNTTQA
MINU
subj
(val)
(lest)
[val)
(dev)
(argl
(test)
Propriciory Madels
GPT-4V (OpeAAL
GPT-lo (OpenAl, 2024)
Gemins-1 S-laash (Team at, 2023)
70.3
75.0
61.6
Gemins-1 Pro (Team al, 20125)
75.0
81.3
57.5
64.0
33.1
69.3
Made' Comparison
LLaVA-OV-..88 al., 2024a)
44.0
43.5
69.2
50.57
3.127
05.8
50.3
32.7"
53.2*
57.2
MM1 (Zhang et al. 2025b)
45.7
56.1
3.14
45.9
71.8
Apello-1.58 (Zohar
63.3
LiVVI-Mipha-1 (Gao
2024)
2-2 (Chen et 2034e)
51.9
54.1
52.0
61.4
37.9
53.4
77.2
Owen VL-2B (Wang et 2024b)
25.6
60.4
53.9
48.7'
62.7
39.4
60.6"
77.2
SF-LLaWA-1:41
529
64.3
39.7
76.7
3B Made' Comparison
Blip3-Vide--B V-4B (Abdin
56.91
2014)
V-Ma 3.1 (Loe 21124a)
45.2
51.7
3.3
43.0
VII A1L.115.30 (Lin al, 2004))
42.2
44.2
49.1
50.90
42.9
44.4
56.1
MM1 5-38 (Zhang et. 20256)
19.5
57.9
3.17
15.4
74.7
LongVU-3 (Shen
55.9
112e (Chen
2024f)
53.0
59.9
71.1'
LinVT-Blip3-BB B-B (Zohar et 2024)
58.3
62.1
58.9
56.6
67.9
59.6
80.1
Apollo 36 (Zobar et al,, 2024)
19.4
60.6
65.0
53.1
68.7
62.5
SF-LLaWA-1.5-0
65.8
55.5
57.3
7B Made' Comparison
VideoChatGI1T-17 (Maaz et 202(b)
35.2
2:12
43.5
VideoLLaVA-22
2023s)
MovieChatt-78 (Songe 2014)
22.5
Pl LaVA-7 (t al., 2014a)
56.3
3.12
40.2'
Tansier78 (Wang et 2024a)
50.5
71.6
LLaVA-NaxiVViden Viden-11 (2hang
20240)
3267
VideoCCha2-HD-3778
2023<)
47.3
VideoLLaMA2-78 (Cheng et al. 2024)
17.9
50.3
514
3.13
18.5
VCMeoCCAM-98 (Fei al., 2024)
53.9
56.1
50.71
63.1
Flash- VStream-BB (Zhang 2024a)
51.9
616
VILA-115-88 al.,
TimeMaker-88 (Chem
2024c)
49.
60
LongVA-7B LoggVa-BB/haag at,
52.6
54.8
3.577
56.3
57.0
69.3
LongVILA-7B (Chen et 2024d)
60.1
65.1
58.1
59.51
57.1
80.7
LongVU-7B (Shen 2l24)
65.4
XComposer-Si SB (Zhang al..
VideolLaMA2211201 (Chem
2024)
54.9
56.4
549
57.4
36.2
56.8
7566
LLVVI-Qwen--VL. (Gao et al., 2024)
63.1
63.3
6011
57.2
68.9
65.8
85.5
MM1 5-7B (Zhang 20256)
53.5
60.9
3.22
29.4
76.9
Kangaroo
2024))
39.4
Orysl (Liu 2025)
70.0
67.5
81.8
1VVA--V-7B at al, 2034a)
58.2
61.5
49.7
56.6
3.5
56.5
64.7
64.2!
79.4
LLaVA-Vido-78 (Zhang al. 2004g)
63.3
69.
65.9
56.51
3.50
58.2
70.8
83.2
Apollo/BB (Zobar al, 2024)
673
58.5
20.9
NVILA--B
20244)
7011
InternVL2.58800 (Chen et al., 20144)
64.2
66.9
60.0
69.0
43.2'
68.31
85.0
Qwen2-VL-78 (Wang et 20244)
63.3
69.0
62.3
55.6"
69.8
447
67 gf
81.2"
SF-LLLVA-115-7B
71.
83.3
Table 10:
Comparison with broader group of Video LLMs on video understanding. denotes reproduced results. *denotes results from the benchmark leaderboard. +denotes results evaluated using GPT-3.5-Turbel 5-Turbo-0613 or an unknown version, which cannot be directly compared with results. Bold and underlined are the best and second-best results for each task.
the image performance, especially on Text-Rich tasks (row and row 2 of each model scale). For reference, 3F..LavAA-1.--mmage- FFF-3-F2FF3BBB outperforms 1---LLLVA-1.5-Imaaeee by +4.9% on OCRBench and +2.7% on TextVQA.
We move on to Stage Il with fully tunable parameters but encounter the oft-of-memory issue (even when we train the 1B model with batch size on H100-80G GPUs). This issue arises from caching a large number of activation values from the visual encoder while extracting features from 128 input frames that is why we do not have this problem in Stage I. Stochastic BP (Cheng et al., 2022) is proposed solve this problem and utilized by modern temporal action detectors (Cheng & Bertasius, 2022) for efficient end-to-end
22

---

Published as a conference paper at COLM 2025
Training Visual Encoder
Knowledge
General VQA
Text-Roe Text-Rich
Model
Stage II
A12D
(test)
SQA
(test)
MMMU
MathV (lestmini)
MM-Vet
RW-QA
OCRBench (teat)
TesVVA (val)
DoeVQA
(test)
1B
Model Comparison
SF-LLaVA-1E Image-E2E E2E-IB
V
73.9
89.3
38.3
53.0
41.1
60.3
74.0
73.8
87.8
ST-LLaVA-L15--maan 5-magge11B
X
20.8
87.8
39.3
512
41.1
57.1
69.5
70.2
85.2
ST-LLaVA-115--225
V
70.5
81.7
38.9
41.7
34.3
55.7
18.6
60.1
68.1
ST-LLaVA-L5-IB
3B Madel Comparison
SF-LLaVA-115 Image 22E-3-E2E-3B E2E-3B
V
77.2
90.0
44.1
61.1
46.0
61.8
27.2
74.7
90.0
33-1LaWA-15-lmage--
75.8
90.0
43.7
57.0
51.1
6188
72.3
72.0
87.5
SF-LL2WA-15-82E-338
SF-LLaVA-1553B
77.0
90.3
44.7
58.6
47.5
63.4
73.4
73.0
83.8
7B Madel Comparison
SF-LLaVA-135 Image F2E.B
V
79.5
91.2
47.1
63.5
47.4
66.9
78.3
75.8
90.7
S1-1L-WA-15-mmaee77
76.7
85.8
44.4
51.0
11.9
60.5
59.6
70.8
78.8
SF-LLaVA-15-778
80.4
91.1
49.0
62.5
54.7
67.5
76.4
76.4
90.3
Table 11:
Results of SF-LLaV--155-222 and VA-LLa VA-.---------mae------ on image bench- marks, which fully train the visual encoder together with the projector and LLM.
Training Visual Encoder
General VideoQA
Long-Form Video Understanding
Temporal Reasoning
Model
VideoMME (w/osub)
PercepTest [val)
LongViceeBench (val)
MLVU (m-avg))
LVBench (avg)
TempComp (mc)
NEXT-QA (test)
Stage
Stage
1B Madel Comparison
SE-IL-VA-15-F2--BB
54.1
58.6
51.5
61.7
59.3
73.9
SF-LLaVA-1.5-1B
56.6
61.9
64.3
39.7
60.5
76.7
3B Model Comptrison
SF-LLaVA-1.5-3B
60.8
65.8
57.3
68.8
43.3
64.0
80.8
7B Made! Comparista
SF-LLaVA-15-122-77 LF-LLAV-15577
59.2
68.1
70.3
44.3
67.9
81.0
63.9
69.6
62.5
71.5
45.3
68.8
83.1
Table 12: Results of SF-LLaVA-1.5-E2E on video benchmarks, which fully trains the visual encoder together with the projector and LLM.
training. However, integrating this mevory-saving technique into multimodal LLMs is non-trivial and is left for future exploration.
Finally, we test tuning the visual encoder only in Stage I and freezing it in Stage II is effective. We train models (named SF-LLaVA-115-E2E) based on SF-LLa VA-1.--lmae- E2E, with the visual encoder frozen. The models are evaluated on both image and video benchmarks, as shown in Table 11 and Table 12. The results show that SF-LLa VA-1.5-E2E performs significantly worse than SF-LLaVA-1E across all metrics. We argue that tuning the visual encoder in Stage harms its generalization ability, leading overfitting on image tasks and conflicts between image and video tasks. We will explore the optimal training strategy for Video LLMs in future work.
A.5 Streaming Video Understanding
Real- Time Visual Understanding
Omni-Source Understanding
Contextual
Model
Understanding
StreamingBench ST-LLaVA-15-7B w/ StreamBridge (Wang at. 2025)
71.1
38.4
35.0
72.5
33.1
36.0
Table 13: Comparison of model performance on different understanding benchmarks.
We convert SlowFast-LLaVV-115577 into proactive streaming assistant by adopting the StreamBridge (Wang et al., 2025) technique without fine-tuning, The results on Streaming- Bench (Lin et al., 2024) are shown in Table 13. We observe that SlowWFastLLLVV-115577 achieves state-of-the-art performance on Real-Time Visual Understanding and Contextual Understanding, showing effectiveness for streaming video understanding. We fall behind on Omni-Source Understanding since our model is never trained on interleaved datasets with audio.
23

---

Published as a conference paper at COLM 2025
B Qualitative Results
Question 1: Describe the video in detail.
SwwFast-LLaVA-1... The video showcases creative drawing process on large white sheet of paper attached to wooden frame. begins with hand holding pink markes sketching a simple geometric shape that evolves house with triangular roof and a door. The hand then adds details like windows and a chimney, followed by a tree with
trunk and branches. A an green marker to color in the clouds, enhancing the scene's vibrancy. The focus shifts to the sun, which is colored blue, and the hand adds yellow details to the tree's branches, creating senee depth and texture. The hand then uses purple marker to add shading to the tree, making it appear more three dimeniinnl.....l.. Throughout background remains consistent, featuring the wooden frame and the large white sheet of paper. The video concludes with the hand adding final touches to the completing vibrant detailed drawing.
drawn
the tree, and clouds addedd above The hand switches
video,
Figure 3: SF-LLa VA-1.5 summarizes a video with detailed caption.
Question According the video, want cite the paper titled 'Slowfest-llvaa, what should do?
NlowFast-LLvV--11 1.5: To cite paper titled stowfast-llava'. you should first search for on Google Scholar. Then, click on the 'Cite' button next to the paper's title. This will open citation dialog box displaying the citation details various formats, such MLA, APA, and Harvard. can choose format you need and copy citation text to use your work.
Question What full the Slowfast-llava paper?
wwwFFat--LLaA-1... The full name of the Slowfast- llava paper is 'Slowfast-llavar A strong training-fre -fre baseline for video large language models."
Question What the author list of the Slowfast-llava paper?
NowFFast-LLaAA The author Xu, Mingze and Gao, Mingfei Gan, Zhe Chen, Hongyou and Lai, Zhengfei and Gang, Haiming and Kang, Kai and Dehghan, Afshin,
Question 4: How many citations does the Slowfast-llava paper have?
Slowfastt-LLVVA-15:: The Slowfast-llava paper 38 citations.
Figure 4: SF-LLaVA-1,, learns the process from the video and captures text-rich details.
the
LwwFFast-LLaV-11.. 1.:: The video showcases child's imaginative play in a toy kitchen setup. A person arranges colorful toy fruita vegetables a wooden countertop, simulating cooking process. They the toys under running water, place them on cutting board, and out them with a toy knife. The person then arranges the cut pieces white plate, adding
toy fruits vegetables, plate back on the countertop, they wash their hands and continue arranging the toy fruits and vegetables. They use black spatula to mix the contents of the pot, ensuring the toy fruits and vegetables well combined. The video stepping away from
wooden spatula
them. After placing the
person hanging black spatula the toy kitchen setup neatly amanged with the toy fruits and vegetables on the countertop.
concludes
the
area, leaving
Question 2: What happens after the person washes the toy fruits?
SlowFast-LLavA- After washing the toy fruits, person arranges them on white plate and then cuts them with toy knife, simulating cooking.
Question 3: How many toy fruits are there the videa?
SlowFast-LLaVA-1.5: The fruits include red strawberry, orange carrot, yellow banana, and purple bunch of grapes.
Figure 5: SF-LLaVA-1.5 understands the relative sequence of different activities.
24


Reviews:
[{"review_id": "7u6ArTcpDT", "reviewer": "Reviewer_2u6V", "summary": "This paper introduces SlowFast-LLaVA-1.5, a family of token-efficient video large language models designed for long-form video understanding. The authors employ the SlowFast mechanism to balance detailed spatial semantics and long-range temporal context, achieving state-of-the-art performance on multiple benchmarks. The streamlined training pipeline and use of publicly available datasets ensure reproducibility. Comprehensive experiments and ablation studies validate the model's effectiveness. The work is well-structured and makes contributions to the field of video understanding with LLMs.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The experiments are comprehensive across 1B, 3B, and 7B model scales, demonstrating the superiority of SlowFast-LLaVA in both image and video understanding. Additionally, the ablation study illustrates that SlowFast achieves a balanced trade-off between performance and efficiency among various projector designs, and also highlights the general effectiveness of architectural design.(GSF, ISF).\n2. This paper provides insights into applying the slow-fast mechanism during training, in contrast to previous works where it was only used at inference time.\n3. The paper is well-written, and easy to follow.", "reasons_to_reject": "1. The paper lacks novelty, as the effectiveness of the slow-fast mechanism has already been demonstrated in prior work [1] on video understanding.\n2. The use of a fixed-frame sampling architecture appears outdated and may be inadequate for addressing the demands of proactive streaming video applications [2].\n3.  The fast way seems minor in table 5, particularly when comparing Row 1 and Row 6 on the LongVideoBench validation set.\n\n[1] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming He. SlowFast Networks for Video Recognition. ICCV 2019.\n\n[2] Junming Lin, Zheng Fang, Chi Chen, Zihao Wan, Fuwen Luo, Peng Li, Yang Liu, Maosong Sun. StreamingBench: Assessing the Gap for MLLMs to Achieve Streaming Video Understanding.  arXiv 2024.", "questions_to_authors": "1. The authors have conducted validation on the Qwen2.5 backbone. It is recommended to further verify the effectiveness of the SlowFast mechanism on a broader range of backbones, such as LLaMA and InterLM.\n2. More benchmarks should be included in Table 5 to better demonstrate the effectiveness of both the slow way and **fast way**.\n3. A more fair comparison policy is recommended for Table 7 when drawing conclusions about the effectiveness of mixed image-video training. Specifically, the 1:1 image-to-video ratio should be compared against the 2:0 (image-only) and 0:2 (video-only) settings.", "extra_scores": {}}, {"review_id": "aIYbxym0px", "reviewer": "Reviewer_9Koa", "summary": "This paper introduces a family of video large language models for long-form video understanding. The authors integrate a SlowFast Video Projector into a LLaVA-style MLLM, and train the model with a simple two-stage training recipe and a carefully curated data mixture. Experiments show the proposed models offer efficiency and strong performance.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The effectiveness of the SlowFast Projector has been verified in the experiments.\n- The paper proposes a simple two-stage training recipe, avoiding complex training stages used in previous works in this field.\n- The authors carefully curate a data mixture based on public data, and the training datasets will be publicly available.\n- The proposed models show strong performance on different model scales, and the model maintains strong image performance.", "reasons_to_reject": "### Major Weakness\n1. The SlowFast Video Projector, which is the main contribution of this paper, is exactly the same as the SlowFast projector proposed in the previous work SlowFast-LLaVA [1]. No architecture modification is applied. The authors experiment with different instantiations of the SlowFast idea termed ISF, but experiments prove that this is not effective, and they used SlowFast-LLaVA's original instantiation. In conclusion, the SlowFast Video Projector should not be the main contribution/innovation of this paper. However, the majority of the paper's content and ablation experiments are about this part.\n2. The curated data mixture is a main contribution. However, the description of this part is quite brief and does not introduce any innovative techniques for data selection or filtering. A more detailed explanation and supporting experiments are needed for this section.\n3. Comparisons between the proposed two-stage training recipe and more complex recipes (e.g., VideoLLaMA 3 [2]) should be included.\n\n### Minor Weaknesses\nThe models utilize SlowFast projector for videos, and MLP projector for images. This design seems to be redundant, as mainstream video mllms (InternVL-2.5 [3], Qwen2.5-VL [4], VideoChat-Flash [5]) can process video and image using a unified projector, by treating image as a single-frame video.  Discussion and experiments about this dual projector design should be included.\n\n> [1] Xu, Mingze, et al. \"Slowfast-llava: A strong training-free baseline for video large language models.\" _arXiv preprint arXiv:2407.15841_ (2024).\n\n> [2] Zhang, Boqiang, et al. \"VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding.\" _arXiv preprint arXiv:2501.13106_ (2025).\n\n> [3]  Chen, Zhe, et al. \"Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling.\" _arXiv preprint arXiv:2412.05271_ (2024).\n\n> [4] Bai, Shuai, et al. \"Qwen2. 5-vl technical report.\" _arXiv preprint arXiv:2502.13923_ (2025).\n\n> [5] Li, Xinhao, et al. \"Videochat-flash: Hierarchical compression for long-context video modeling.\" _arXiv preprint arXiv:2501.00574_ (2024).", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0NhVgcVRZp", "reviewer": "Reviewer_sRib", "summary": "The paper introduces SlowFast-LLaVA-1.5, a family of token-efficient VLMs for long-form video understanding. It incorporates the SlowFast mechanism into a streamlined training pipeline, using only publicly available datasets. The models, ranging from 1B to 7B parameters, achieve state-of-the-art performance on various video and image benchmarks, demonstrating superior efficiency and reproducibility.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The proposed VLMs have shown state-of-the-art performance on various video and image benchmarks.\n\n2. The proposed VLMs achieve high efficiency, achieving high performance with only 1B and 3B parameters.\n\n3. The presentation and organization of the paper are clear and easy to follow.", "reasons_to_reject": "I do not have significant concerns that can lead to the rejection of this paper.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MPhlwE6dQ1", "reviewer": "Reviewer_edor", "summary": "This paper presents a well-motivated and technically sound advance in the domain of video LLMs, targeting long-form video understanding with an emphasis on token efficiency. The paper introduces a family of models at scales ranging from 1B to 7B parameters that integrate the two-stream Slow-Fast mechanism into a LLaVA-style architecture, enabling the models to process both spatially detailed frames at a low rate (Slow pathway) and motion-rich frames at a high rate (Fast pathway) in a token-efficient manner. Training is carried out in two straightforward stages—first on image-only data for general knowledge and reasoning, and then on a joint image-video mixture drawn exclusively from publicly available datasets—ensuring both reproducibility and competitive performance. Empirical results on benchmarks such as LongVideoBench (62.5% at 7B) and MLVU (71.5% at 7B) demonstrate that even the smaller 1B and 3B variants outperform or match existing methods, particularly in long-form scenarios, while using substantially fewer tokens per video input.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work addresses a pressing challenge in multimodal AI—how to build Video LLMs that are both capable of understanding hour-long content and efficient enough for edge deployment. By systematically incorporating the Slow-Fast projector into a unified supervised fine-tuning pipeline, the authors strike a judicious balance between frame count and token complexity, yielding clear empirical gains across general videoQA, long-form understanding, and temporal reasoning benchmarks. The two-stage training recipe is refreshingly simple, eschewing proprietary data and elaborate multi-phase schedules in favor of publicly accessible datasets and a modestly tuned LLM backbone. The comprehensive ablation studies—ranging from design choices of Slow–Fast instantiations to video–image data ratios—underscore the robustness of the approach and offer practical guidance for future research. Importantly, the open‑source release of model weights and data mixtures will foster reproducibility and community adoption, amplifying the paper’s impact on both academic and industrial fronts.", "reasons_to_reject": "Despite its strengths, the paper leaves several concerns unaddressed. First, the reliance on uniform FPS sampling when videos exceed 128 frames risks overlooking semantically critical segments in longer recordings, potentially biasing model outputs; the proposed mitigation of adding timestamps remains speculative without empirical validation. Second, by freezing the visual encoder throughout training, the models may underexploit domain-specific visual features that could further boost performance, particularly on text‑rich or fine‑grained visual tasks. Third, while the Slow-Fast mechanism delivers efficiency, its comparative performance against emerging memory‑augmented or dynamic token‑merging techniques is not explored; this narrows the paper’s experimental scope. Finally, the presentation, though generally clear, is dense with acronyms and heavy math in the architecture section, which may impede accessibility for readers less familiar with prior LLaVA and SlowFast literature.", "questions_to_authors": "* Have you considered augmenting the uniform frame sampler with a learned importance function or keyframe detector to address the limitation that “a ten‑minute video and a one‑hour video have the same number of input frames”? Incorporating a lightweight attention or reinforcement‑learning module could dynamically prioritize salient segments.\n* What are the inference latency and memory footprints of the 1B and 3B models on a representative edge device (e.g., a smartphone GPU)? Detailed profiling would substantiate claims of mobile‑friendliness.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces SlowFast-LLaVA-1.5 (SF-LLaVA-1.5), a state-of-the-art family of Video Large Language Models tailored for token-efficient long-form video understanding. The authors aim to balance computational efficiency with detailed spatial and temporal modeling through the integration of the SlowFast mechanism into a streamlined training pipeline.

Pros:
- The authors report comprehensive results application to various video and image understanding benchmarks.
- The integration of the two-stream SlowFast mechanism is seen as a key factor in improving long-form video understanding while maintaining high efficiency.

Cons:
- The concerns of uniform FPS sampling of missing key frames.
- The authors may add more discussion comparing with the previous SlowFast architecture, which is helpful for readers to understand the motivation of this paper.

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept