Improving LLM Video Understanding with 16 Frames Per Second
Authors: Yixuan Li
Changli Tang
Jimin Zhuang
Yudong Yang
Guangzhi Sun
Wei Li
Zejun MA
Chao Zhang
Venue: ICML 2025 poster (2025)
Keywords: Multi-modal large language models
high-frame-rate video understanding
video LLM
Primary area: applications->computer_vision

Abstract: Human vision is dynamic and continuous. However, in video understanding with multimodal large language models (LLMs), existing methods primarily rely on static features extracted from images sampled at a fixed low frame rate of frame-per-second (FPS) $\leqslant$2, leading to critical visual information loss. In this paper, we introduce F-16, the first multimodal LLM designed for high-frame-rate video understanding. By increasing the frame rate to 16 FPS and compressing visual tokens within each 1-second clip, F-16 efficiently captures dynamic visual features while preserving key semantic information.
Experimental results demonstrate that higher frame rates considerably enhance video understanding across multiple benchmarks, providing a new approach to improving video LLMs beyond scaling model size or training data. F-16 achieves state-of-the-art performance among 7-billion-parameter video LLMs on both general and fine-grained video understanding benchmarks, such as Video-MME and TemporalBench. Furthermore, F-16 excels in complex spatiotemporal tasks, including high-speed sports analysis (*e.g.*, basketball, football, gymnastics, and diving), outperforming SOTA proprietary visual models like GPT-4o and Gemini-1.5-pro.
Additionally, we introduce a novel decoding method for F-16 that enables highly efficient low-frame-rate inference without requiring model retraining. We will release the source code, model checkpoints, and data at [https://github.com/bytedance/F-16](https://github.com/bytedance/F-16).

Improving LLM Video Understanding with 16 Frames Per Second
Yixuan Li Changli Tang" 1
Jimin Zhuang' Yudong Yang' Chao Zhang'
Guangzhi Sun?
Wei Li Zejun Ma²
Abstract
1. Introduction
Large language models (LLMs) have demonstrated excep- tional performance across many natural language process- ing tasks, with some even nearing human-level performance (OpenAI et al., 2024; Dubey et al., 2024; Touvron et al., 2023; Du et al., 2022; Bai et al., 2023). The impressive abil- ity of LLMs to understand, generate, and reason with text has sparked significant interest among researchers, drawing attention from both academia and industry to expand their capabilities multimodal understanding.
Human vision is dynamic and continuous. How- ever, in video understanding with multimodal large language models (LLMs), existing meth- ods primarily rely on static features extracted from images sampled at a fixed low frame rate of wrame-pe--second (FPS) <2, leading to critical visual information loss. In this paper, we intro- duce F-16, the first multimodal L.M designed for high-frame-rate video understanding. By in- creasing the frame rate to 16 FPS and compress- ing visual tokens within each 1-second clip, F- 16 efficiently captures dynamic visual features while preserving key semantic information. Ex- perimental results demonstrate that higher frame rates considerably enhance video understanding across multiple benchmarks, providing new ap- proach improving video LLMs beyond scaling model size or training data. F-16 achieves state-of- the-art performance among 7-billion- --billion-parameterr video LLMs on both general and fine-grained video understanding benchmarks, such as Video- MME and TemporalBench. Furthermore, F-16 excels in complex spatiotemporal tasks, includ- ing high-speed sports analysis (egg., basketball, football, gymnastics, and diving), outperform- ing SOTA proprietary visual models like GPT-4o and Gemini-1.5-pa Additionally, introduce novel decoding method for F-16 that enables highly efficient low-frame-rate inference with- out requiring model retraining. We will release the source code, model checkpoints, and data https//giihub..oo/yytedda
In video understanding, videos contain both slowly chang- ing elements, such as backgrounds and scenes, and rapidly changing, fleeting details, such as body movements and micro- expressions. Given the high frame count in videos, processing every frame computationally expensive. As result, existing video LLMs (Li et al., 2024; Zhang et al., 2024b; Wang al., 2024; Lin et al., 2024b; Cheng et al., 2024; Tang et al., 2024: Yao al., 2024) primarily focus on slowly changing elements, typically employing low-frame- rate sampling or selecting a fixed number of frames, effec- tively treating every 1-second video clip as a few images. However, this approach has a critical limitation: while low- frame-rate sampling like I Trame-per-secee sufficient for capturing video's overall theme and context, it fails to preserve rapidly changing visual cues, leading to information loss. This significantly hinders the model's ability to comprehend dynamic scenes, thereby limiting its overall video understanding capabilities.
(FPS) may be
To address these limitations, this paper introduces F-16, a video LLM designed for more human- -ike video perception by processing coherent video frames at higher frame rate of 16 FPS. A key challenge in frae--rame-rate video under- standing is the increased visual token sequence length and redundant information across consecutive frames. To miti- gate this, we propose a visual-text aligner that not only trans- forms visual features into text-like tokens suitable for the I.M backbone but also compresses redundant intra-frame and inter-frame information within each I-second video clip. To leverage the strengths of existing high-peeformance image encoders, we adopt a 3-layer multi-layer perception (MLP) as the aligner, extending a pre-trained image LLM to process the dynamic features in 16 FPS videos while
"Equal contribution 'Tsinghua University BBytDannce. Corre- spondence to: Chao Zhang <c2277@ @siinghua..uuunn....
Proceedings of the nrnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Improving LLM Video Understanding with 16 Frames Per Second
preserving rich semantic features from each static frame. Experimental results demonstrate that F-16 achieves state- of-the-art (SOTA) performance among models of similar sizes and remains competitive with much larger models on general video cersstiion-nnwering (QA) benchmarks, includ- ing Video-MME (Fu et al., 2024). MLVU (Zhou et al., 2024), Long VideoBennh (Wu et al., 2024), and NeXT-QA (Xiao et al.. 2022). as well as fine-grained video understanding benchmarks like TemporalBench (Cai et al., 2024) and Mo- tionBench (Hong et al., 2025). The superior performance of F-16 highlights the advantage of reducing visual informa- tion loss through 16 FPS sampling, Additionally, after fine- tuning on high-speed sports tasks such as basketball, foot- ball, gymnastics, and diving, F-16 significantly outperforms SOTA proprictary models like GPT-4o and Gemini- Finally, we introduce a training-free coding method, allowing F-16 to adapt to lower-frame- frame-rate scenarios without increasing test-time computational costs.
ple at low frame rate select a fixed number of frames, treating them separate images with orders. In many ap- proaches, each frame is first encoded using pre-trained image encoder, then mapped to the text space via a modality aligner, and finally fed into an L.M backbone for response generation. Video-LLaVA (Lin et al., 2024a) uniformly samples 8 frames from a video, processes each frame inde- pendently through an image encoder, and generates video tokens that are then passed to the LLM. LLaVA-One Vision (Li et al., 2024) is designed to handle single images, multi- ple images, and videos. For videos, it samples frames at I FPS and reduces the number of video tokens using bilinear interpolation. Building on a similar architecture, Zhang et al. develops a strong video-understanddng LLM with syn- thetic data, but still limits sampling to 1 FPS. Qwen2-VL (Wang al., 2024) increases the frame rate employs rotary position embedding to enhance temporal modelling. Meanwhile, Videol LMM 2 (Cheng et al., 2024) and Video-SALMONN 2 (Tang et al., 2024) incorporate full audio information to support video understanding. However, VidcoLLaMA only processes 16 frames from each video, while SALOON uses no more than 30 frames.
de-
FPS and
Our main contributions are summarized as follows:
The first high-fraan high-frame-rate video LLM: We developed F- 16, the first video LLM capable of perceiving videos at 16 FPS. F-16 supports videos up to 1 10 seconds, processing 1,760 visual frames per video, and achieves SOTA per- formance on multiple video understanding benchmarks among 7-billion (B)-parameter video LLMs.
Low frame rate sampling can result in critical visual infor- mation loss, particularly in videos with rapidly changing scenes, intricate details, or fast motion. Additionally, if keyframes are missed, yet the model is trained on labels that rely on keyframe information, it may struggle to align its predictions with the expected content, potentially leading to hallucinations and degraded performance. To address these challenges, Koala (Tan al., 2024), Frame Voyager (Yu et al., 2024b), and VeyVideLLLM (Liang et al., 2024) employ intelligent sampling strategies to extract key infor- mation more effectively. Mcanwhile, MA-LMM (He et al., 2024) and VideoStreaming (Qian et al., 2024) explore denser video sampling to enhance model performance.
High-frame-rate sports video benchmarking: To eval- uate video LLM comprehension in high-frame-rate see- narios, we curated dataset of sports videos that require fine- fine-grained temporal understanding. F-16 significantly outperforms GPT-4o and Gemini-1.5-Pre on these tasks.
decoding: We propose variable-frame-ree rrae decoding method that enables F-16, trained at 16 FPS, to be seamlessly applied to scenarios suitable for low-frame-rate videos, reducing computa- tional cost without hurting the performance.
Efficient pariable-frame-ree
2.2. Temporal Input Compression in Video LLM
More frames must be sampled to enable high-frame-rate video understanding, making temporal information com- pression between adjacent frames essential for efficient com- putation. A common approach aggregates image features extracted from each frame to compress redundant informa- tion effectively. RLT (Rafailov et al., 2024) reduces the total number of video tokens by dropping redundant patches, identified through frame differentials. Video-LaVIT (Jin et al., 2024) integrates keyframes and motion vectors as input and trains a motion encoder to enhance motion under- standing. Espresso (Yu et al., 2024a) employs specialized spatial and temporal Q-poolers and compressors to elim- inate redundant information. NVILA (Liu et al., 2024c) follows a "scale-then ten-tthhencomppesss paradigm, first scaling up both frame resolution and frame count, then applying spatial linear compression and temporal averaging to reduce the
2. Related Work
2.1. From Image LLM to Video LLM
Most image LLMs adopt the approach of connecting an image encoder to an LLM via modality adapters, achieving remarkable success. LLaVA (Liu et al., 2024b;a) applies instruction tuning (Wei et al., 2022), enabling zero-shot image understanding. BLIP-2 (Li et al., 2023) integrates frozen image encoder with an LLM using Q-Former to bridge the modalities. In (Chen et al., 2023) further enhances accuracy by scaling up the visual encoder for more precise image representations. Intern VI. 2.5 (Chen et al., 2024) delves into model scaling and achieves better test-time performance.
Regarding video understanding, recent studies typically sam-
2

---

Improving LLM Video Understanding with 16 Frames Per Second
total number of visual tokens. Notably, these approaches mostly focus on processing long videos efficiently rather than processing videos with high frame rates.
using Eqns. (3) and (4) as:
H, = ((GELU(
(3)
3. Methods
To further compress the visual tokens, spatial 2 x 2 max pooling function Max2DPool()) is employed after the aligner, as shown in Eqn. (4). Specifically, H2 is first re- sized to RvPx /pxh, then Max2DPool(.) is applied on the first and second dimension, which results in about 4 times reduction of the total number of visual tokens. Note that
3.1. Model Architecture
The overall architecture of F-16 is illustrated in Fig. 1. F-16 takes video and text prompt as the inputs and generates textual responses accordingly.
For the input video, frames are first sampled at a high frame rate, indicating the sampled frames are adjacent to each other and they may be visually very similar. Denote F, as the i ith frame sampled from the video and n as the total number of sampled frames. Each frame is encoded by the pre-trained image encoder Enc(-) as
H = (ax)))ool(HH),
(4)
where H2 E R are the visual tokens of the thh processing window and are fed to the backbone LLM.
Z, = (F;), Osi<n.
(1)
Finally, visual tokens of all non-overlapped processing win- dows form the final visual token sequence H of the whole video. The backbone LLM is required to generate a textual response Y given the user's text instructions I and the visual token sequence H by
Here, Z, E RRxd is the visual features of F, output by the image encoder, where p is the number of patches and d is the output dimension of the image encoder.
Given the high similarity between adjacent frames, we aim to compress image features within a local time window in the visual feature space, preserving overall visual semantics while effectively capturing the dynamic features introduced by temporal changes across frames. Previous studies (Liu et al., 2024b:c) suggest that linear transformations outper- form nonlinear ones in mapping image features to the LLM input space, as they better retain the semantic information extracted by the image encoder. Building on these insights, we design our high-frame-rate aligner using a 3-layer MLP, which consists of two linear layers with a GELU (Hendrycks & Gimpel, 2016) activation function in between.
Y maxy P)Y
(5)
3.2. Building F-16 by Extending Image LLM
Video LLMs are typically built on well-trained image LLMs to leverage their pre-trained visual perception abilities. How- ever, key challenge arises as shown in Fig. 1: the dimen- sions of the high-frame-rate aligner's 's modules P(.) and Q() do not match those of the single-image aligner in the image LLM when extending an image LLM to video LLM. This mismatch prevents direct parameter initialization using the pre-trained image L.M,, as illustrated in Fig. 2. To address this, we adopt the block matrix decomposition approach, breaking down F-16's nigh-dimensional modality aligner into smaller sub-matrices that can be initialized using pa- rameters from the image LLM,, ensuring a smoother transfer of pre-trained knowledge.
Let w be the number of frames in processing window, and let Zjw: Zjw+1: ..., (j++)w=-1 represent the visual features of these w frames in the j th window, and P(-) and Q(-) as the first and second linear layers of the high-frame-ree rrate aligner, respectively. To construct the input to the aligner, the visual features of all frames within the processing win- dow are first concatenated along the feature dimension as:
Specifically, F-16 extends a pre-trained image LLM with LLaVA-One Vision structure (Li et al., 2024), which has single-frame aligner with two linear layers as shown in Fig. 2(a). The aligner takes angle-image features as input and outputs the corresponding visual tokens. Denote its first and second linear layers (-) and B(-), and their weight matrices and bias vectors as WA E Rdxh, WB E Rlixh, ba E Rt, and bB E R², respectively. Similarly, denote the weight matrices and bias vectors of the two linear layers P(.) and Q(-) of the high-frame-ratt aligner in F-16 as WP E Rudxwh Wq E RwhxA, bp E Rwh, and bq e
0ed = Concat(Zzar Zjw+1,.... 1yj+1)
(2)
where Z E Rd is the combined visual features of the j th window.
Next, the ligh-rramerrate aligner maps 7 to the input dimension of the LLM, denoted as h. Specifically, the first linear layer P--) maps the features from the dimension of wd to wh, while the second linear layer Q(-) maps from wh to h. Therefore, the output vectors H, E R of the nigh-frame-rate aligner for the j th window can be obtained
3

---

Improving LLM Video Understanding with 16 Frames Per Second
No.
Yes
LLM
LLM
<video> Does the shot get in?
<video> Does the shot get in?
Single-famee Aligner
High-frame-rate Aligner
Image Encoder
Image Encoder
Sparse Sampling
Dense Sampling
Input Video
Input Video
Low- fra-fra---rate modeling
High- rate modeling
Figure 1. The overall architecture of F-16. Compared the classical structure of low-framc-rate video LLMs, F-16 samples more frames with a higher frame rate of 16 FPS and uses a high-frame-rate aligner, which not only preserves the overall visual semantics but also extracts the dynamic features from the changes across frames, without introducing more visual tokens.
RR. F-16's parameters can be initialized using
H, = Q(GELU(P((Z""))
WB
T
GELU (AAU(AAZjuu (Zjw))
WA
A
O
WB
GELU(A (Zne+1))
O
WA
A
+ +B
WP
(6)
W
WB
GELU (A ((
O
WA
ba
WB B
B(GELU (AELLUAAZZAAAT)) (Z)x+))))
(9)
ba
1
WB
bp
bQ bB:
(7)
w
To improve training stability, in our implementation, set the off-diagonal elements in the Wp to random noise initialized with Kaiming Uniform (He et al., 2015), rather than all zeros.
ba
WB
where O is a zero matrix. This initialization leads to the fact that F-16's aligner's output is the average the representa- tions of each frame in its initial state, since Eqn. (3) can be expanded using Eqns. (8)-(9)
3.3. Variable-Frame-Be Frame-Rate Decoding
Encoding each frame with an image encoder significantly increases the computational load, making inference slower compared to low-frame-rid models. This becomes a major bottleneck for real-time processing. To address this, we introduce variable-frame-rade decoding method, allowing F-16 to perform low-frame-ra inference, thereby reducing computational costs while maintaining strong performance.
Z ue Z
ba
WA
O
Zju+1
ba
P(Z
+
O WA
:
Z6+1)=-11
ba
A (Z)
At test-time, assume that we want to reduce the frame rate by factor of k. compared to the frame rate used for training. The processing window can be correspondingly reduced to w/k: To meet the input dimension of the high-frame-rate
A AZ w+1)
(8)
A A (Z4))))41))

---

Improving LLM Video Understanding with 16 Frames Per Second
model on high-speed sports videos. Videos for gymnastics, diving, basketball, and foothall are collected for further tun- ing, where FineGym (Shao et al., 2020), Diving48 (Li et al., 2018), SoccerNet (Giancola et al.. 2018), and NBA video clips are used respectively. To demonstrate the importance of high-frame-rate modeling in sports videos, the visualiza- tion for a video randomly sampled from Diving488 is shown in Appendix D.
1
GELU
GELU
P
Regarding the FineGym (Shao et al., 2020) data for gymnas- ties understanding, we sample 90% clips as the training set while the remaining 10% as the test set and ensure that the duration of videos in the training and test sets is balanced. Video captioning and open-ended QA of gymnastics are trained, and open-ended QA is evaluated. Accuracy the used evaluation metric.
(a) Single-frame Aligner (b)
Aligner
Figure 2. The comparison between (a) single-frame aligner and (b) high-frame-He fea---rrmrrttteee rate aligner. With more input frames, the high- frame-rate aligner concatenates the frames and uses linear layers P.)) and Q() that have larger input dimensions to compress and encode the frames to visual tokens.
Regarding the Diving48 (Li et al., 2018) data for diving understanding, we use its official data split. In addition, the diving actions are split into four phases including takeoff, somersault, twist, and flight, following the original paper. Formatted captions describing all the phases are used to train models. The mean accuracy of predicting actions of the four phases is used as the evaluation metric.
aligner, the features of each frame need to be repeated k times. Under this condition, in the j th processing window, Eqn. (2) can be expressed as follows:
The "Ball Action" subset of SoccerNet (Giancola al., 2018) is used for football understanding. Counting the times of ball passes in the video is our focus task, since it is essential to use high frame-rate information. We segment the videos into 10-second clips and count the number of passes in each segment to provide training labels. Accuracy serves as the evaluation metric.
Z Jj jw/k Concat Z ju/k/ fk: Zjm/ks Z /zn
A times
= = Concat(ZZ jw/ jw/k/
ZJ .... Z' 215+1)m/kk11)
Therefore, z E Rexud, which still meets the input di- mension requirements of the high-frame-raad aligner.
Regarding the NBA data, we collected NBA matches from 2024/11/13 to 2024/1 1/25, with total number of 276 matches. We focus on the task of whether the ball goes into the net, which requires high-frame-rate information. We evaluated the performance using the test set of NSVA (Wu al., 2022) using Fl score. We manually annotated video captions of 10,000 NBA video clips. These video captions focus on details of players' movements on the field, the movement of the ball, and other aspects of movement, providing the model with high-frame-rre data annotations. Examples of these sports data can be found in Appendix A.
4. Experimental Setup
4.1. Model Specifications
F-16 utilizes the LLaVA-OV model of LLLVA--OneVisee (Li et al., 2024) with 7B parameters based image LLM, which uses SigLIP (Zhai et al., 2023) as the visual encoder and Qwen2-7B (Yang et al., 2024) as the backbone LLM.
Besides, F-16 samples frames from the video at 16 FPS as input, allowing for maximum of 16 x 110 = 1760 frames. For videos longer than 110 seconds, F-16 uniformly samples 1760 frames from the video input. The width of the processing window is set equal to the FPS as w= 16.
4.3. Training Specifications
For general video training, the high- fatb--frae--atee aligner and the LLM are updated, while the image encoder stays frozen. F-16 is trained for epoch on the training data using 128 H100 GPUs, with a learning rate set x 10 5.
4.2. Data Specifications
The training data of general videos are the same as VL-
Video (Zhang et al., 2024b), including VL-VVooo-do-178 (Zhang et al., 2024b), I.LVVA-Hound (Zhang et al., 2024a), NEXT-QA (Xiao et al., 2021), Aetivit (Yu et al., 2019) and Perception (Patraucean et al., 2024).
For further tuning the model on high-speed sports data, LoRA (Hu et al., 2022) is adapted to the LLM and serves as the only trainable module in this stage. The rank and the scaling factor of LoRA are set to 128 and 2.0, respectively. We tnne F-16 using 64 H100 GPUs for 5 epochs, with
Besides generie video understanding, we also fine-tune the
5

---

Improving LLM Video Understanding with 16 Frames Per Second
Table Comparison of video QA results of different video L.M.s. The FPS maximum number of frames sampled by the model listed in the table. We evaluate the models on Video-MME, Viseo (VST), TemporalBench (TPB), MotionBench (MB), NeXT-QA (NQA), MLVU, LongVia Vidh (LVB) for comprehensive performance comparison. &Accuracy the evaluation metric, except for TPB, which uses % Mullliip Binary Accuracy (Cai et al., 2024) for evaluation. F-16 achieves SOTA on Video-MME, NQA, TPB, and MB among all models and offers competitive results compared SOTA proprietary models like GPT-4o and Gemini-
Video-MME
NQA 1
TPB f
MB f
VST MIVU1 f
LVB 1
Model
FPS #Frame
Avg-t
St
M1
L1
Test
Short
Val.
Avg.
m-Avg.
Val.
GPT-4o
IFPS / 384
71.9
80.0
70.3
65.3
38.0
33.0
78.3
54.5
66.7
Gemini-1.5-Pro
IFPS
75.0
81.7
74.3
67.4
26.6
51.0
64.0
Qwen2-VL-7B
2FPS / 768
63.3
24.7
52.0
75.6
55.6
Vid20LL
16
47.9
60.5
48.4
VideoChatt-HHD-7B
16
45.3
79.5
37.4
39.3
LLaVA-OV-7B
32
58.2
79.4
21.2
73.0
64.7
56.3
MiniCPM-V22.-8B
64
60.9
71.3
59.4
51.8
21.4
52.0
54.9
LLaVA-Video-70
64
63.3
83.2
22.9
70.8
58.2
NVILA-7B
1024
64.2
75.7
62.2
54.8
82.2
70.1
57.7
F-16-7B (ours)
16FPS / 1760
65.0
78.9
63.2
52.8
84.1
37.2
54.5
74.4
70.3
57.6
learning rate set to 2 x 10-5.
commercial models like Gemini- (emini-1.5-Ppp 1.5-Pro and GPT-4o.
5. Experimental Results
5.2. High-Speed Sports Video Understanding
In this section, explore the high-speed sports video under- standing of our models. We use the sports data of gymnas- tics, diving, basketball, and football to fine-tune our general video understanding models trained using FPS = 1 and FPS = 16. GPT-4o and Gemini- 1.5-Pro are also evaluated by uniformly sampled 120 frames as its input, to provide an upper bound on the existing Video LLMs on these tasks. Note that both models cannot recognize gymnasties and div- ing actions, as these two scenarios require much expertise. The detailed results are shown in Table 2.
5.1. General Video Understanding
Results of the general video understanding are presented in Table 1. F-16 achieves the SOTA results among the 7B models in several video QA benchmarks, including not only the general video QA benchmarks Video-MME and NeXT-QA, but also the fine-grained vidco understanding benchmarks TemporalBench and MotionBench.
On the Video-MME Short, Medium, and NeXT-QA datasets-each designed for short video understanding- model surpasses the previous 7B SOTA model by 3.2%, 1.0%, and 0.9% in accuracy, highlighting its strong per- formance short videos. For benchmarks evaluating long video understanding. such as Video-MME Long, Beng and MLVU, the challenge is greater due to sparser frame sampling, causing frames within the pro- cessing window to exhibit more significant variations. This increases the difficulty for the modality aligner to effectively encode temporal changes within the limited token represen- tation. As result, F-16 experiences slight performance drop compared to LLa LLLVVA-Vide--71 (Zhang et al., 2024b), which is trained on the same video dataset.
Table 2. Results of high-speed sports understanding. F-16 with the high-fran high-frame-rate aligner outperforms the low-frame-rate aligner version in all sports tasks. We also evaluate GPT-4o and Gemini- 1.5-Pro on the NBA and SoccerNet QAs, which do not require the knowledge from the in-domain training to answer the questions.
Gym
Diving
NBA
Soccer
Model
&Acc.t
AAcc.
%Flf
Acc..1
GPT-4o
76.7
36.8
Gemini-1.5-PP F-16 FPS
80.6
43.1
48.5
76.0
87.1
55.4
F-16 FPS = 16
64.1
86.5
92.9
57.7
Results on TemporalBench and MotionBench which focus on motion and temporal understanding, show a significant positive effect of high-frame-rate modeling. F-16 gains an improvement of 13.5% and 2.5% respectively on the two benchmarks compared with the existing 7B SOTA mod- els and also achieves competitive performance of SOTA
The results demonstrate that high-frame-rza model out- performs both the low-frame-rate model and commercial models across all four sports tasks, highlighting the impor- tance of temporal perception. Among these tasks, high- frame-rate modelling shows the most significant advantage in gymnastics and diving, improving accuracy by over 15%
6

---

Improving LLM Video Understanding with 16 Frames Per Second
and 10%, respectively. This likely because accurately dis- tinguishing movements requires capturing the full motion sequence, rather than relying on sparse frames. For example, in gymnastics, determining whether a movement is clock- wise or counterclockwise can be difficult with only sparsely sampled frames. Similarly, in diving, assessing the degree of rotation or twist is challenging without a continuous mo- tion view. In basketball and football, while the advantage of migh-frame-rate modelling is smaller, it remains notable. This is likely because the model can leverage additional con- textual cues, such as player behaviour, audience reactions, or score changes, to infer outcomes even in the absence of keyframes. For instance, whether a shot was successful or a pass occurred.
6.0
5.0
Inference Time
4.0
LLM (max token)
LLM (max 1024 tokens)
3.0
Encoder
2.0
High-fiame-rate Aligner
1.0
0.0
Test FPS
(a) Time consumption of different modules of F-16 when inference. Models are prompted to do video captioning on Video-MNN MMM Long (with 300 vidcos) with different test FPS and different max lengths.
Beyond QA, our model also demonstrates strong high- frame-rate captioning capabilities. Even advanced models like GPT-4 struggle with accurate captioning of high-speed sports videos. Appendix C presents comparison of sports captioning outputs across different models, showcasing the effectiveness of our approach.
65.0
Train FPS-Test FPS
Vidco-MME Ace (%)
Truin FPS-16
64.5
64.0
5.3. Variable-Frame-Ree Rate Testing
63.5
In high-frame-rate modeling, since each frame needs to be encoded using the encoder, this can significantly impact the model's s inference speed. Testing at a frame rate lower than the training frame rate can increase speed with mini- mal performance loss. Fig. 3 shows the performance and testing. The infer-
63.0
2
16
Test FPS
inference time of suriable-frame-ree ence is considerably slowed down by the image encoder when the test FPS increases, as Fig. 3a shows. Especially when outputting only a very small number of tokens, such as when answering questions, the time taken by the image encoder in gigh-framerrate modelling even surpasses that of the backbone LLM.
(b) Video-MM MME performance of models trained and tested at dif- ferent FPS. The solid line indicates that the model trained and tested using the same FPS, while the dashed line indicates the model is trained at FPS = 16 and tested at lower FPS.
Figure 3. Performance and inference time analysis of variable- frame-rate testing.
fariable-frame-ratt decoding proves be an effective lution for addressing this challenge. As shown in Fig. 3b, adapting the high-frame-rate model to low-frame-rate in- ference achieves comparable performance on generic video benchmarks like Video-MME, with only slight degrada- tion compared to evaluation at FPS = 16. Moreover, it performs favourably against models trained at the same test FPS. This flexibility allows developers to balance compu- tational efficiency and high-frame-rate advantages based on specific tasks. For example, lower FPS can be used for fast reasoning in general video comprehension, while higher FPS is preferable for fine-grained understanding in high-speed video analysis tasks.
high-frame-rate details. We compare the cosine similarity dews, between visual features across frames, as shown in Table 3. Additionally, we apply a 2 x 2 Max2DPool() function to the visual features and compute the cosine sim- ilarity after pooling, denoted daus,ater. impact of this basic compression method on the feature rep- resentations. As expected, cosine similarity before pooling decreases progressively as frames shift over time, eventually dropping to 0.5898 by the 4th frame, indicating clear differ- ences in visual features. However, after max pooling, the cosine similarity remains high (0.8828) at Frame 4, despite significant changes in the scene and player actions. This suggests that fine-grained visual features not dominant compared to global features and are largely suppressed by max pooling. Based on this observation, we hypothesize that preserving the complete visual features as input to the figh-frame-rate aligner can lead to significant performance improvements, as retains fine-graing temporal details
observe the
5.4. Analysis on High-Frame Rate Aligner
To gain deeper insights into the mechanism of the high- frame-rate aligner, we first analyze the visual features out- put by the image encoder to verify the presence of subtle
7

---

Improving LLM Video Understanding with 16 Frames Per Second
Table average cosine similarity all image tokens between frames and the reference frame. deos, before refers the cosine distance before pooling, while deas, aner refers to the cosine distance after pooling.
Reference Frame
Frame
Frame 2
Frame
Frame 4
deos, befure deteratter
0.8945 0.9609
0.7656
0,6680
0.5898
0.9180
0,9023
0.8828
This trimming approach effectively leverages the local pa- rameters of the high-fram righ-frame-rate aligner for low-frame-rate decoding, distinguishing it from the repeating- frame method introduced in Section 3.3. To evaluate its effectiveness, we compare trimming with frame repetition for oww-rramerrate inference, denoted as "Trimming" and "Repeat", respec- tively. The results, presented in Table 5, highlight the differ- ences between these two methods.
Table Results general video data using different pooling strategics for models at different frame rates.
Video-MME
FPS
Pooling
Avg.1
St
M1
L
pre
62.3
76.6
59.7
50.6
1
post
62.9
77.7
60.3
50.8
16
pre
60.8
75.0
58.2
49.2
16
post
65.0
78.9
63.2
52.8
decoding results Video-MME us- ing different methods. "Trimming" represents only using the sub- matrices of the original aligner, and "Repeat" represents repeating frames to meet the input dimension of the original aligner. The normal decoding results are also provided.
Table S.
essential for high-speed video understanding.
We conduct experiments on general video data using dif- ferent pooling strategies across models with varying frame rates. Specifically, examine pre-pooling, where max pooling is applied before the modality aligner, and post- pooling, where max pooling is applied after the modality aligner. The results are presented in Table 4. For the FPS= model, post-pon post-pooling provides only a marginal improvement of less than 1 1 on the Video-MME. benchmark. However, for the FPS= 16 model, pre-pooling proves even less ef- fective than in the FPS= 1 scenario, whereas post-pool post-pooling yields notable 4.2% improvement. These findings suggest that the high-frame-rate aligner heavily relies on inter-frame variations in the visual features to learn effectively. Pre- pooling prematurely removes fine- mne-grained temporal details, making harder for the frame-rate aligner, with its larger parameter set, to extract complex motion patterns, potentially leading to overfitting and degraded performance. In contrast, post-pooling preserves these temporal details, thereby enhancing the model's overall performance.
Video-MME
Method
FPS
Avg.t
St
M
L1
16
65.0
78.9
63.2
52.8
Trimming
8
62.3
76.7
61.0
49.2
Repeat
8
63.9
78.0
62.4
51.4
The "Trimming" decoding method results in performance decline, whereas the "Repeat" method nearly preserves the high-frame-rate performance. This suggests that the high- frame-rate aligner does not simply perform an averaging operation in the feature space, despite being initialized that way. Instead, effectively extracts motion information be- tween semantic features through its 3-layer MLP structure, rather than relying solely local parameter adjustments.
Additionally, we experimented with more structures for the high-frame-rate aligner, but the results were unsatisfactory. This further confirms that using linear mappings or MLPs as aligners is more effective preserving the semantic space of visual features. Detailed results of these experiments are provided in Appendix B.
Additionally, we conducted another way to reduce the test FPS to see how the high-frame-rate aligner works. The parameters WP Rudxwh, bp E Rdth Wq e Rwhxx bQ R of the aligner are trimmed subma- trices according to the test FPS in Eqns. (10)-(11):
6. Conclusions
This work presents F-16, a powerful video LLM designed for 16 FPS video understanding, along with the high -rame- rate aligner. To the best of our knowledge, this the first
W Wpl: sd, : sh], bpp b sh]
(10)
W2 Wal: sh, :). bq2 bq-
(11)
8

---

Improving LLM Video Understanding with 16 Frames Per Second
study explore high-frame-rate processing in video LLMs. We also introduce method initialize video LLMS from pre-trained image LLMs, enabling F-16 to leverage their rich pre-trained knowledge for enhanced video understand- ing. Additionally, we propose a variable-frame-rate decod- ing method, which balances high- frah-fram--rtte performance and inference cost without requiring additional training. As result, F-16 achieves SOTA performance in general video QA among models of similar size and demonstrates clear advantage in high- frame-rae-ree video understanding, outper- forming commercial models such GPT-4o. This work opens new directions for advancing high- -rae-rram--rtte video comprehension in multimodal LLM research.
3 Herd of Models. arXiv preprint arXiv:2407:21783, 2024.
Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S.,. Zhang, R., Wang.
Z.,, Zhou, C., Shen, Y., Zhang, M., al. Video-MME:
The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis. arXiv preprint arXiv:2405.21075. 2024.
Giancola, S., Amine, M., Dghaily, T., and Ghanem, B. Soc- cerNet: A scalable dataset for action spotting in soccer videos. In Proc. CVPR Workshops, Salt Lake City, 2018.
He, B., Li, H., Jang, Y. K., Jia, M., Cao, X., Shah, A., Shrivastava, A., and Lim, S.-N. MA-LMM: Memory- Augmented Large Multimodal Model for Long-Term Video Understanding. In Proc. CVPR, Seattle, 2024.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
He, K., Zhang, X., Ren. S., and Sun, J. Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification. In Proc. ICCV, Santiago, 2015.
Hendrycks, D. and Gimpel, K. Gaussian Error Linear Units (GELUs). arXiv preprint 1606.06600.000 08415, 2016.
References
Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., et al. Qwen Tech- nical Report. arXiv preprint arXiv:2309. 2023.
Hong, W., Cheng, Y., Yang, Z., Wang, W., Wang, L.,, Gu, X., Huang, S., Dong, Y., and Tang, J. MotionBench: Benchmarking and improving fine-grained video motion understanding for vision language models. arXiv preprint arXiv:2501.02955, 2025.
Cai, M., Tan, R., Zhang, J. Zou, B., Zhang, K., Yao, F,, Zhu, F., Gu, J.. Zhong, Y., Shang, Y., et al. Tempo- ralbench: Benchmarking fine- grained temporal under- standing for multimodal video models. arXiv preprint MxX1v:2410.10814 2024.
Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L.,, and Chen, W. LoRA: Low-Ran Adaptation of Large Language Models. In Proc. ICLR, 2022.
Chen, Z., Wu, J., Wang, W. Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., Li, B., Luo, P., Lu, T., Qiao, Y., and Dai, J. Intern VI. Scaling up vision foundation models and aligning for generic visual- linguistic tasks. arXiv preprini arXiv:2312. 142.23888, 2023.
Jin, Y., Sun, Z., Xu, K., Chen, L., Jiang, H., Huang, Q., Song, C., Liu, Y., ZHANG, D., Song, Y., al. Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual- Tokenization. In Proc. ICML, Vienna, 2024.
14238,
Chen, Z., Wang, W., Cao, Y., Liu, Y., Gao, Z., Cui, E., Zhu, J., Ye, S., Tian, H., Liu, Z., et al. Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling. arXiv preprint arXiv:2412.05271, 2024.
Li, B., Zhang, Y., Guo, D.,, Zhang, R., Li, F., Zhang, H., Zhang. K., Zhang, P., Li, Y., Liu, Z., et al. Llava- OneVision: Easy visual task transfer. arXiv preprint UrXiv:2408.033266 2024.
Cheng, Z., Leng, S., Zhang, H., Xin, Y., Li, X., Chen, G., Zhu, Y. Zhang, W., Luo, Z., Zhao, D., and Bing, L. VidcoLLaMA 2: Advancing Spatial-Tempone Modeling and Audio Understanding in Video-LLMMs arXiv preprint arXiv:2406.07476, 2024.
Li, J., Li, D., Savarese, S., and Hoi, S. BLIP-2: Boot- strapping language- pre-training with frozen image encoders and large language models. In Proc. ICML, Honolulu, 2023.
Li, Y., Li, Y., and Vasconcelos, N. RESOUND: Towards action recognition without representation bias. In Proc. ECCV, Munich, 2018
Du, Z., Qian, Y., Liu, X., Ding, M., Qiu, J., Yang, Z., and Tang, J. GLM: General language model pretraining with autoregressive blank infilling. In Proc. ACL, Dublin, 2022.
Liang, H., Li, J., Bai, T., Huang, X., Sun, L., Wang, Z., He, C., Cui, B., Chen, C., and Zhang, W. KeyVideLLLMM: Towards Large-scale Video Keyframe Selection. CoRR, 2024.
Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., et al. The LLLMMA

---

Improving LLM Video Understanding with 16 Frames Per Second
Lin, B., Zhu, B., Ye, Y., Ning, M., Jin, P., and Yuan, L. Video-LLaVA: Learning united visual representation by alignment before projection. In Proc. CVPR, Seattle, 2024a.
Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., et al. Qwen2-VIL VL: Enhanc- ing Vision-language Model's Perception of The World at Any Resolution. arXiv preprint arXiv:2409. 12191, 2024.
Lin, J., Yin, H., Ping, W., Molchanov, P., Shoeybi, M., and Han, S. VILA: On pre-training for visual language models. In Proc. CVPR, Seattle, 2024b.
Wei, J., Bosma, M., Zhao, V., Guu, K., Yu, A. W., Lester, B., Du, N., Dai, A. M., and Le, Q. V. Finetuned Language Models Zero-Shot Learners. In Proc. ICML, 2022.
Liu, H., Li, C., Li, Y., and Lee, Y. J. Improved baselines with visual instruction tuning. In Proc. CVPR, Seattle, 2024a.
Wu, D., Zhao, H., Bao, X., and Wildes, R. P. Sports Video Analysis on Large-Scale Data. Proc. ECCV, Tel Aviv, 2022.
Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruction tuning. In Proc. NeurlPS, Vancouver, 2024b.
Wu, H., Li, D., Chen, B., and Li, J. Long A benchmark for long-context interleaved video-language understanding. In Proc. NeurlPS, Vancouver, 2024.
Liu, Z., Zhu, L., Shi, B., Zhang, Z.,, Lou, Y., Yang. S., Xi, H., Cao, S., Gu, Y., Li, D., al. NVILA: Efficient Frontier Vi- sual Language Models. arXiv preprint arXiv:241..04468, 2024c.
Xiao, J., Shang, X., Yao, A., and Chua, T.-S. NEXT-QA: Next phase of question-answering to explaining temporal actions. In Proc. CVPR, Nashville, 2021.
OpenAI, Achiam, J., Adler, S., Agarwal, S., Ahmad, L.. et al. GPT-4 Technical Report. arXiv preprint arXiv:2303.08774, 2024.
Yang, A., Yang. B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C.,, Li, C., Liu, D., Huang, F, et al. Qwen2 Technical Report. arXiv preprint arXiv:2407. 10671, 2024.
Patraucean, V., Smaira, L.., Gupta, A., Recasens, A., Mar- keeva, L., Banarse, D., Koppula, S., Malinowski, M., Yang, Y., Doersch, C., et al. Perception Test: A Diagnos- tic Benchmark for Multimodal Video Models. In Proc. NeurlPS, Vancouver, 2024.
Yao, Y., Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et al. MiniCPM-V: A GPT-4V Level MLLM on Your Phone. arXiv preprint 0rXiv:2408.0/800 2024.
Yu, K. P., Dave, A., Ambrus, R., and Mercat, J. Espresso: High Compression For Rich Extraction From Videos for Your Vision-Language Model. arXiv preprint arXiv:2412.04729, 2024a.
Qian, R., Dong, X., Zhang, P., Zang, Y., Ding, S., Lin, D., and Wang, Streaming Long Video Understand- ing with Large Language Models. arXiv preprint arXiv:2405. 160009940./6600, 2024.
Yu, S., Jin, C.. Wang, H.,, Chen, Z., Jin, S., Zuo, Z., Xu, X., Sun, Z., Zhang, B., Wu, J., et al. Frame- Voyager: Learning to Query Frames for Video Large Language Models. arXiv preprint arXiv:2410.03226, 2024b.
Rafailov, R., Sharma. A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. Don't Look Twice: Faster Video Transformers with Run-Length Tokenization. In Proc. NeurlPS, Vancouver, 2024.
Shao, D., Zhao, Y., Dai, B., and Lin, D. FineGym: A hierar- chical video dataset for fine-gra -rained action understanding. In Proc. CVPR, Seattle, 2020.
Yu, Z.,, Xu, D., Yu, J., Yu, T., Zhao, Z., Zhuang, Y., and Tao, D. Activitynet-QA: A dataset for understanding complex web videos via question answering. In Proc. AAAI, Honolulu, 2019.
Tan, R., Sun, X., Hu, Wang, J.-h., Deilamsalehy, H., Plummer, B. A., Russell, B., and Saenko, K. Koala: Key frame-condine canee-coniiiooee long video-LLM. In Proc. CVPR, Seattle, 2024.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid Loss for Language Image Pre-Training. arXiv preprint 2rXiv:2303.1534. 2023.
Tang, C., Li, Y.. Yang. Y.. Zhuang, J., Sun, G., Li, W., Ma, Z., and Zhang, C. Enhancing multimodal LLM for detailed and accurate video captioning using multi-round preference optimization. arXiv preprint 100.:2240 2024.
Zhang, R., Gui, L., Sun, Z., Feng, Y., Xu, K., Zhang, Y., Fu, D., Li, C., Hauptmann, A., Bisk, Y., et al. Di- rect Preference Optimization of Video Large Multimodal Models from Language Model Reward. arXiv preprint 002500058..., 2024a.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., et al. LLaMA: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2002. 13971, 2023.
Zhang, Y., Wu, J., Li, W., Li, B., Ma, Z., Liu, Z., and Li, Video Instruction Tuning with Synthetic Data. arXiv preprint arxiv:2410.02713.. 2024b.
10

---

Improving LLM Video Understanding with 16 Frames Per Second
Zhou, J., Shu, Y., Zhao, B., Wu, B., Xiao, S., Yang, X., Xiong, Y., Zhang, B., Huang, T., and Liu, Z. MLVU: A comprehensive benchmark for multi-task long video understanding, arXiv preprint 2rXiv:2406.04266 2024.

---

Q: Please describe this diving video. A: The diver executed back take-off transitioned into a Improving LLM Video Understanding with 16 Frames Per Second
A. Examples of the Sports Video Data
Table 6, Examples of sports data, including Cym, Diving, Soccer and NBA.
Sports Type
Category
Content
QA
Q: Please describe the gym video between 0.0s and 5.7s in detail, The player performs "round-off, flic-flac on, stretched salto backward with turn off".
Gym
Caption
Please descrihe the gym video in detail and pay attention the time. A: The video is sport video about Floor Exercise. The player first performs "double salto backward piked" between 19.7s and 21,5s. Then, the player performs "2 turn or more with heel of free leg forward at horizontal throughout turn" between 28.0s and 29.4s. Then, the player performs "salto backward stretched with 2.5 twist" between 43.3s and 44.3s. Then, the player performs "switch leap ring position" between 51.2s and 52.1s. Then, the player performs "split laap with turn" between 52.9s and 53.9s. Then, the player performs "salto forward tucked" between 64.1s and 64.9 9s. Then, the player performs "salto backward stretched with 2 twist" between 66.2s and 67 .s. Then, the player performs "switch lapp with turn" between 71.3s and 72.5s. Then, the player performs "double salto backward tucked" between 91.0s and 93.0s.
Q:
QA
Q: What type of take off did the diver use? A: Reverse.
Diving
Caption
15 somersaults, and incorporated 5 twist while in free position.
Soccer
QA
Q: How many football passes are there in the video? A: 2.
QA
Q: Did the shot go in? A: Yes.
NBA
Caption
Q: Explain the NBA vidco clip thoroughly, ensuring flecting details are captured. A: Hawks player No. 2 Wallace tossed the ball to No. 1 Johnson on the left side of the threc-point line. Johnson jumped up from the left side of the basket and caught the ball with his right hand. That's 's when the Celtics' No. 7 tried stop him. Johnson made two center-foot stance-up moves and made feint pass to the No. 10 from beyond the three-point line. At I 1::5, No. 10 made a three-point shot in front left of the rim before being blocked by Celtics player No. 0 T'atumm After being blocked, the number 10 grabs the ball in the same position and dribbles once, passing the ball to the number on the left baseline of the frame.
Table. 6 shows the examples for each category of the sports video used in training and evaluation that are mentioned in Sec. 4.2.
B. Different Modality Aligners for High-Frame-
Understanding
The high- frame-rate aligner mentioned in Sec. 3 uses linear layers that compress the frames temporally as the main layers and form an MLP. However, more structures are tried in our experiments, as shown in Table. 7-9. Linear is finally selected based on the experimental results.
Table 7. Comparison between linear and CNN methods.
Video-MME
Structure
FPS
Max Time
LLM Training
Avg.t
St
Mt
L
Linear
32
30
LoRA
61.5
75.0
59.2
50.3
2D-CNN
32
30
LoRA
57.7
69.1
55.9
48.2
3D-CNN
32
30
LoRA
57.2
70.0
54.8
46.9
Table. 7 shows the comparison between linear and CNN methods, which implement spatiotemporal convolution to replace the linear layer and the pooling layer, in order to make the model compress and understand spatiotemporal information together. 2D-CNN conducts convolution on the temporal dimension and spatial visual token dimension together, while 3D-CNN resizes the spatial dimension into a 2D graph like the operation before pooling in Sec. 3. However, the CNN models fail to understand fine-grained video details and achieve worse final results. The results show that spatiotemporal
12

---

Improving LLM Video Understanding with 16 Frames Per Second
compression and encoding difficult to conduct in one step. Besides, the CNN models get quite different structure compared to the aligner in image models, meaning that it cannot be easily initialized with the original pretrained aligner and requires much more training costs to conduct image pretraining.
Table 8. Comparison between linear and dual linear method.
Video-MME
Structure
FPS
Max Time
LLM Training
Avg.t
St
M
L.
Linear
32
30
LoRA
61.5
75.0
59.2
50.3
Dual Linear
32
30
LoRA
58.8
72.3
55.8
48.2
Table. 8 shows the comparison between linear and dual linear methods. Dual linear replaces the original max pooling layer with learnable linear layer, aiming to learn better spatial compression. Though the model performs better in training loss, it performs worse in the test stage, showing worse generalization ability. That's probably due to the limitation in the scale of video data, making the learned spatial compression linear slightly overfits the training set. Meanwhile, the spatial linear layer is difficult to initialize with the modules in image LLMs or human-tuned parameters, leading to the waste the rich knowledge image LLMs.
Table 9. Comparison between linear and attention method.
Video-MME
Structure
FPS
Max Time
LLM Training
Avg.1
S
M
Lt
Linear
32
30
LoRA
61.5
75.0
59.2
50.3
Attention
32
30
LoRA
61.8
74.2
60.3
50.9
Linear
16
60
Full
63.5
78.1
60.6
51.8
Attention
16
60
Full
62.9
76.8
60.2
51.9
Table. 8 shows the comparison between linear and attention methods. Attention similar structures to linear, but replaces the first linear layer with self-attention layer, to further catch the connection of the frames. It is observed that attention does achieve better results when training LoRA A only, but fails to maintain the advantage when training full LLM. The difference is probably brought by the minimal transform of attention that fits the original pretrained LLM better (each frame independently processed with Wy matrix in aiff-attention) but does not bring more advantages when all the parameters are trainable. Meanwhile, the quadratic complexity of the attention mechanism also makes it harder to extend to a higher max time. In our implementation, due to the VRAM limitation, it cannot be extended 10s with 16 FPS like the linear one, which also reduces the potential of attention aligner.
C. Captioning Cases for High-Speed Sport Videos
We show detailed NBA captions generated by models trained at different frame rates in Table 10. The groundtruth caption and GPT-4o's caption are also provided. GPT-4o and the FPS = 1 model misses ball-passing information or describes incorrectly, while the high-frame-rate model able to eatch these details.
D. Visualization of Sports Videos at Different FPS
We randomly sample a video from Diving48 and show the sampled frames at different FPS in Fig. 4-6.
13

---

Improving LLM Video Understanding with 16 Frames Per Second
Table 10. Captioning Cases for 10-second NBA video of different models.
Text Type
Content
Groundtruth
Jazz player No. 23 held the ball outside the left three- three-point line, guarded by Lakers player No. 46. He suddenly passed the ball to Jazz player No. 40, a teammate cutting into the paint from the three-point line in the middle. The latter attempted layup at 10:25, and Lakers player No. 3 raised his right hand, trying block the shot. The ball didn't i go in. Lakers player No. 15 grabbed the rehound and passed to player Lakers player No. Lakers player No. then dribbled the ball past half-court.
GPT-4o (Inputted with 120 frames)
The NBA video clip captures a sequence from a game between Utah Jazz and an opposing team. The play begins with the Jazz on offense, setting up a play on the perimeter. The player with the ball attempts to drive towards the basket but is met with strong defense from the opposing team. The ball handler then passes This player makes a move to penetrate the defense ball is then passed around the perimeter as the Jazz look for an open shot. Eventually, player attempts shot near the basket but misses. The opposing team secures the rebound and quickly transitions into fast break, pushing the ball up the court. The clip ends with the opposing team advancing towards their offensive end, looking capitalize the transition opportunity.
teammate who is positioned near the three-point line. is also challenged by defenders. The
Generated by model trained at FPS 1
The No. 17 player of the Jazz held the ball in the backcourt. He dribbled once with his left hand, then switched his right hand, and moved towards the frontcourt. After three more dribbles, entered the paint. Then he jumped up and threw the ball towards the basket at 12:24. The baskethall hit the rim and hounced out. The No. player of the Lakers jumped up and grabhed the rebound with both hands.
Generated by model trained at FPS 16
Jazz player No. 24 dribbled the hall once outside the three-point line on the left side of the basket and passed to Jazz player No. 15 who moved to the right side of the basket outside the three-point line. Jazz player No. 15 took jump shot 10:23, but failed score. Lakers player No. 3 jumped up and raised his right hand to try to block shot. The basketball hit the rim and bounced out. Then Jazz player No. 0 grabbed the rebound and passed the ball to Jazz player No. 24 who moved to the right side of the basket outside the three-point line.
Lonson 2012
Figure 4. Visualization at FPS=1
Figure 5. Visualization FPS=4
14

---

Improving LLM Video Understanding with 16 Frames Per Second
000
Opp
Figure 6, Visualization at FPS=16
15


Reviews:
[{"review_id": "7tJv9iBEva", "reviewer": "Reviewer_nuva", "summary": "This paper proposes a new method F-16 that increases the frame rate of existing video LLM to 16 frames per second (FPS). The paper argues that existing video LLMs, which typically operate on low frame rates (e.g., 1 FPS), lose crucial dynamic visual information. F-16 aims to address this by processing videos at a significantly higher frame rate while employing a visual-text aligner to compress redundant visual information within 1-second clips. This allows the model to capture subtle but important motion cues. The paper claims that F-16 achieves state-of-the-art performance among 7B parameter video LLMs on both general (Video-MME, TemporalBench) and fine-grained video understanding benchmarks. Further, it excels in complex spatiotemporal tasks such as high-speed sports analysis, outperforming models like GPT-4o and Gemini 1.5 Pro. The authors also propose a novel decoding method enabling efficient low-frame-rate inference without retraining the model.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful and constructive feedback on our paper. Below, we will respond to the questions you have raised. \n\n---\n\n**1: Can you provide more detailed information about the training procedure, including hyperparameter settings, training time, and the resources (e.g., number of GPUs) used for training?**\n\nRegarding the training procedure, F-16 is first initialized from LLaVA-OneVision-7B, as shown in Sec. 4.1. Then, F-16 is trained on general videos. At this stage, the image encoder stays frozen while other parts of the model are updated. To further verify the advantages of high-frame-rate modeling, the model is then fine-tuned on high-speed sports video. LoRA is adapted to the LLM and serves as the only trainable module in this stage.\n\nAs for other training settings, F-16 is trained with Adam optimizer and cosine scheduler. The learning rate is set to 2e-5, the warm-up ratio is set to 0.03, and the batch size per device is 1. For general video training, 128 H100 GPUs are used for training for 1 epoch (about 13000 steps of updates), which takes about 35 hours. As for high-speed sports fine-tuning, 64 H100 GPUs are used for training 5 epochs (about 9000 steps of updates), which takes about 20 hours. For comparison, the FPS=1 model takes about 18 hours for general video training and 10 hours for high-speed sports fine-tuning.\n\nNote that the difference in training time primarily comes from two parts: a small portion is due to the increase in encoder duration, while the major part is the catastrophic growth in CPU time when reading more frames in long videos, which leads to less GPU usage. We have not optimized video frame extraction, and just use the Python Library\"Decord\" for frame extraction. Appropriate optimization or preprocessing can significantly improve training speed, such as extracting frames in advance. \n\n---\n\n**2: What are the computational costs (memory footprint, inference time) of F-16 compared to other video LLMs? This information is important to understand the practical limitations of the proposed approach.**\n\nWith the high-frame-rate aligner, though 16 times more visual frames are inputted, the tokens into the LLM remain the same as the low-frame-rate model. Therefore, the computational cost for LLM is the same as compared of the low-frame-rate one. The difference mainly comes from the encoder and the aligner. \n- Inference time. As compared in Fig. 3(a), the inference time of the proposed high-frame-aligner can be ignored compared with the LLM and the visual encoder, and the comparison to other models can mainly focus on the visual encoder. To our observation, the inference time for the visual encoder grows linearly with the input frame time. Therefore, the inference time of the visual encoder increases by 16x, which makes the total inference time of F-16 1x longer than that of the comparable video LLM with FPS=1. \n- Memory footprint. The memory cost can be divided into 4 parts: the final visual tokens, the hidden layer of the aligner, the output of the visual encoder, and the inner computation cost of the visual encoder. The final visual tokens are kept at the same number, which does not increase in memory cost. For the other 3 parts, though it seems that they will grow linearly along with the number of frames, they can be handled sequentially because of the independence between different processing windows. For instance, using a for-loop to process frames between processing windows. Therefore, there is no significant increase in memory cost observed from F-16.", "claims_and_evidence": "The claims made in the submission are generally well-supported by evidence.\n\nClaim: Higher frame rates enhance video understanding. This is supported by the consistent performance gains observed across various benchmarks (Video-MME, TemporalBench, MotionBench, sports datasets) when comparing F-16 with other 7B models.\n\nClaim: F-16 excels in high-speed sports analysis. Table 2 demonstrates a significant performance advantage of F-16 in tasks like gymnastics, diving, basketball, and football compared to other video LLMs. The comparison to GPT-4o and Gemini 1.5 Pro is also compelling.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria seem reasonable for the problem of video understanding.\n\nHigh-frame-rate sampling: The decision to use 16 FPS is empirically motivated and aligns with the need to capture more dynamic visual information. The claim that this is a better trade-off is reasonable.\n\nVisual-text aligner with 3-layer MLP: The proposed aligner is designed to transform visual features to text like tokens. The MLP design is justified for its efficacy and compatibility with existing image encoders.\n\nEvaluation benchmarks: The use of standard video understanding benchmarks like Video-MME, TemporalBench, MotionBench, and sports datasets is appropriate for evaluating the performance of F-16. The choice of metrics (Accuracy, F1-score) is also standard for the tasks considered.", "theoretical_claims": "There are no theoretical claims in the paper that require proof checking.", "experimental_designs_or_analyses": "The experimental designs and analyses seem sound.\n\nComparison with other models: The paper compares F-16 with several state-of-the-art video LLMs and proprietary models (GPT-4o, Gemini 1.5 Pro) to demonstrate its effectiveness.\n\nAblation studies: The ablation studies on pooling strategies and alignment strategies (Table 4 and Table 5) provide insights into the importance of different components of F-16.\n\nAnalysis of high-frame-rate aligner: The analysis of visual features output by the image encoder and the cosine similarity analysis helps to understand the benefits of the high-frame-rate aligner.", "relation_to_broader_scientific_literature": "The paper builds upon the existing literature on video LLMs, particularly those that focus on improving video understanding by leveraging pre-trained image encoders and modality alignment techniques. The key contribution of this paper is the emphasis on higher frame rates, which has been relatively unexplored in the context of LLMs. The paper acknowledges and compares its approach with existing methods for video processing and temporal input compression.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths:\n\n-Novelty: The focus on high-frame-rate video understanding is a novel and important direction for video LLMs.\n\n-Performance: F-16 achieves state-of-the-art performance among 7B models.\n\n-Efficiency: The variable-frame-rate decoding method enables efficient inference without retraining.\n\nCompleteness: The paper is well-written and provides sufficient details about the proposed methods and experimental results.\n\nWeaknesses:\n\nWhile variable frame rate decoding is discussed, the actual cost and memory footprint requirements are not shown, this is a limiting factor.", "other_comments_or_suggestions": "The paper could benefit from a more detailed analysis of the computational cost of F-16, including the memory footprint and inference time.\n\nIt would be interesting to explore the potential of using even higher frame rates (e.g., 30 FPS) and the trade-offs between performance and computational cost.", "questions_for_authors": "Can you provide more detailed information about the training procedure, including hyperparameter settings, training time, and the resources (e.g., number of GPUs) used for training? This information would help to better assess the reproducibility of the results.\n\nIf the training process is relatively efficient, it would make the paper more appealing to the wider community.\n\nWhat are the computational costs (memory footprint, inference time) of F-16 compared to other video LLMs? This information is important to understand the practical limitations of the proposed approach.\n\nIf F-16 has a manageable computational cost, it would strengthen the paper's claim of being a practical solution for high-frame-rate video understanding.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZOPalQUHuU", "reviewer": "Reviewer_xf2Z", "summary": "This paper studies the problem of high-frame-rate video understanding. The authors claimed that existing methods for video understanding merely sample video frames at a low FPS (mostly lower than 2), where there exists critical information loss. To tackle this problem, they introduce F-16, a novel multimodal large language model (MLLM) specially designed for high-frame-rate video understanding at 16 FPS. The main contributions are summarized as: 1) [model]: The first high-frame-rate video LLM, 2) [Benchmark]: A High-frame-rate sports video benchmark, and 3) [method]: Efficient variable-frame-rate decoding.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive rating for the paper. We sincerely appreciate your recognition of our work.", "claims_and_evidence": "The main claim of the paper is that \"the existing paradigm for video understanding (sampling at around 2FPS) is sub-optimal, which would lose much information when facing highly dynamic videos\". This claim is clear, reasonable, and supported with convincing evidence.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "There are no proofs in the submission.", "experimental_designs_or_analyses": "The experiments are conducted on both the proposed high-frame-rate benchmark (NBA videos) with manual annotations and public video understanding benchmarks. The experiment protocols are reasonable.", "relation_to_broader_scientific_literature": "The key contribution is to extend the existing Video-LLMs from a low frame rate to a high frame rate, which is more natural and more suitable for analyzing highly dynamic videos. To my best knowledge, this is the first work to do such exploration.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Generally, this is a good paper about LLM-based video understanding. Extending existing paradigms to 16FPS is a non-trivial setting. The proposed high-frame-rate aligner can well-balance the efficiency and performance.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "fFxueVfvzT", "reviewer": "Reviewer_g4TB", "summary": "The paper explores high FPS in video understanding with MLLM. It is an interesting and meaningful attempt and the authors employ some techniques to solve the problem of excessive number of tokens. The performance gain is promising in some specialized scenarios like sports as expected.\n\n## update after rebuttal\n\nI appreciate the author's response, but I still have concern on the temporal design of pre-fusing 16 or other number of frames within one second. The current temporal design equals to decoupled visual temporal perception module with a LLM. The utilization of high frame rate input is restricted by the temporal fusion module. A smarter token selection rather than fusion enables more systematic VLM for processing high frame rate videos.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your thoughtful and constructive feedback on our paper. Below, we will respond to the questions you have raised.\n\n---\n\n1: **The evaluation on existing general video benchmarks cannot showcase the advantage of high frame-rate training. & Supplement more scenarios beyond sports**\n\nIn fact, in general video understanding, F-16 also demonstrates significant advantages in short video benchmarks. Note that F-16 uses less data (less video data and image data) compared to most of the models in Table 1, and it only uses LLaVA-Video-178k. Besides, as for more scenarios, we also evaluate F-16 on a very recent benchmark, FAVOR-Bench, which targets evaluating **fine-grained video motion understanding**. F-16 achieves SOTA results  on it among 7B models. \n\nWe list the results on these benchmarks here to more clearly present the advantages of F-16:\n\n|                  | Video-MME Short | NExT-QA            | TemporalBench   | MotionBench     | FAVOR-Bench        |\n| ---------------- | --------------- | ------------------ | --------------- | --------------- | ------------------ |\n| Previous 7B SOTA | 75.7 (NVILA)    | 83.2 (LLaVA-Video) | 24.7 (Qwen2-VL) | 52.0 (Qwen2-VL) | 41.5 (VideoLLaMA3) |\n| F-16             | **78.9**        | **84.1**           | **37.2**        | **54.5**        | **46.0**           |\n\nFor long video understanding, although F-16 is not outstanding, it still shows competitive results. This is because only when processing short videos, the frame sampling is done at FPS = 16, matching the training situation. For long videos, the FPS is much lower than 16. For instance, as F-16 samples a maximum of $110\\times 16=1760$ Frames, it perceives a 1760-second video at FPS=1. Sparse frame sampling will lead to slightly poorer performance of the high-frame-rate aligner, resulting in average results for long videos.\n\n---\n\n**2: More technical designs and experiments on the aligner.**\n\nWe studied various high-frame-rate modeling structures. All attempts confirmed that visual-language alignment via linear transformations helps prevent model performance decline, which means linear projections effectively and efficiently align the visual encoder's output semantic space with the LLM's input space. This is consistent with prior work like NVILA.\n\nAppendix B details the studied structures. Using CNN modules to capture frame spatiotemporal differences led to worse results than MLP, suggesting CNN aligners may harm semantic information derived from the visual encoder. Replacing max pooling with a learnable linear layer improved training loss but is worse when testing. Using a self-attention layer to replace the MLP projector's first linear layer to extract frame dynamic changes also affected visual feature semantics, resulting in slightly worse performance when scaling up the training parameters.\n\nUltimately, we chose a 2-layer MLP structure. Its first linear layer ensures semantic alignment between LLM input and visual encoder output spaces, and the second compresses duplicated information in continuous frames. It is well-known that any continuous mapping function can be represented arbitrarily accurately by a 2-layer MLP with sufficiently large hidden layer dimension, which provides an insight into our motivation. Experimental results also validate this design.\n\n---\n\n**3: More analysis on the aligner is desired, including input temporal range, output number of tokens, inserting positions, etc.**\n\nAs for the temporal range, we have trained models at different FPS, and set the width $w$ of the processing window equal to the FPS. As the input FPS gradually increases from 1 to 16, the performance of the model shows an upward trend, shown in Fig. 3(b) \"Train FPS=Test FPS\". Besides training, testing at different FPS is also tried, shown in Fig. 3(b) \"Train FPS=16\".\n\nThough the input number of tokens to the aligner increases with FPS, the output number of tokens remains the same due to the processing window width of the aligner remains the same with FPS. This setting enables a fair comparison of models with different frame rates.\n\nFor inserting positions, since the high-frame-rate aligner is a replacement to the single-frame aligner, only pre- and post-pooling used by other VLLMs are evaluated. Table 3 shows pre-pooling causes larger differences between adjacent frames, and Table 4 shows post-pooling performs much better under high-frame-rate settings. These indicate that the high-frame-rate aligner heavily relies on inter-frame variations in visual features for effective learning.  If adjacent sampled frames are no longer similar to each other, the high-frame-rate aligner will struggle to learn well.\n\n---\n\n**4: Present the visualizations of model inputs under different frame rates.**\n\nWe visualize videos at different frame rates here: https://github.com/F-16-LLM/Rebuttal/blob/main/README.md\n\nUnder low-frame-rate cases like FPS = 1, many details will be missed. We will add this part to the updated paper.", "claims_and_evidence": "The claims are supported by the experimental results.", "methods_and_evaluation_criteria": "1. The method of merging high frame-rate tokens is very simple, utilizing the generalization from single-frame to multiple-frame scenarios. However, this strategy has limited temporal capacity and cannot fully leverage the temporal dynamics in the high frame-rate input, slightly superior to average pooling. Either heuristic token preprocessing or more systematic model design is expected.\n2. The evaluation on existing general video benchmarks cannot showcase the advantage of high frame-rate training, so the authors extend to the high-speed sports scenarios. It is necessary to supplement more scenarios beyond sports, compare with more video specialist models and conduct more comprehensive ablations on the high frame-rate token processing.", "theoretical_claims": "Correct", "experimental_designs_or_analyses": "More analysis on the aligner is desired, including input temporal range, output number of tokens, inserting positions, etc.", "relation_to_broader_scientific_literature": "The high frame rate exploration is a significant part that pushes VLM to achieve human-level perception, which can be related with human perception literatures.", "essential_references_not_discussed": "No missing references", "other_strengths_and_weaknesses": "The problem is meaningful but the authors should study more technical designs on the aligner and conduct more comprehensive experiments.", "other_comments_or_suggestions": "None", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The manuscript received ratings of 4, 3, and 2. Reviewers generally appreciated the idea, impressive performance of the proposed approach while maintaining the efficiency, and papre writing. Reviewers also raised several questions in the initial review including, additional details about the training procedure (e.g., hyperparameter settings, training time, and the resources),  input temporal range, and output number of tokens. Post-rebuttal, two reviewers remained positive while the third reviewer still has concerns on the temporal design of pre-fusing a number of frames. Authors presented the required additional details in the rebuttal, including the training procedure such as, hyperparameter settings, training time, and the resources (e.g., number of GPUs) used for model training. Given the reviews and the rebuttal, the AC agrees with reviewer nuva and reviewer xf2Z that the main contribution of extending the existing video-LLMs from a low frame rate to a high frame rate is interesting and is well supported with reasonable experimental protocols, demonstrating consistent performance gains observed across various benchmarks. The altnertaive idea of exploring a smarter token selection, as suggested by reviewer g4TB is also worth exploring, but the authors showed that their proposed approach based on high FPS in video understanding with MLLM while employing a visual-text aligner to compress redundant visual information, is simple and leads to performance gains (also acknowledged by reviewers).
Decision: Accept (poster)