VideoEval: Comprehensive Benchmark Suite for Low-Cost Evaluation of Video Foundation Model

## Abstract

Abstract With the growth of high-quality data and advancement in visual pre-training paradigms, Video Foundation Models (VFMs) have made significant progress recently, demonstrating their remarkable performance on traditional video understanding benchmarks. However, the existing benchmarks (e.g. Kinetics) and their evaluation protocols are often limited by relatively poor diversity, high evaluation costs, and saturated performance metrics. In this paper, we build a comprehensive benchmark suite to address these issues, namely VideoEval . Specifically, we establish the Vid eo T ask A daption B enchmark ( VidTAB ) and the Vid eo E mbedding B enchmark ( VidEB ) from two perspectives: evaluating the task adaptability of VFMs under few-shot conditions and assessing their representation power by directly applying to downstream tasks. With VideoEval, we conduct a large-scale study on 20 popular open-source vision foundation models. Our study reveals some insightful findings on VFMs: 1) overall, current VFMs exhibit weak generalization across diverse tasks, 2) increasing video data, whether labeled or weakly-labeled video-text pairs, does not necessarily improve task performance, 3) the effectiveness of some pre-training paradigms may not be fully validated in previous benchmarks, and 4) combining different pre-training paradigms can help improve the generalization capabilities. We believe this study serves as an important complement to the current evaluation for VFMs and offers valuable insights for the future research.

## 1 Introduction

Figure: Figure 1: Overview of VideoEval. We propose a novel, vision-centric evaluation method for video foundation models that is comprehensive, challenging, indicative, and low-cost.
Refer to caption: x1.png

The field of deep learning is experiencing a paradigm shift of pre-training and transfer learning due to the emergence of foundation models (FMs). These models, exemplified by BERT [], GPT [, , ], CLIP [] and Stable Diffusion [], are pre-trained on massive and diverse data at scale and demonstrate remarkable generalization power to a broad spectrum of downstream tasks.

In the realm of video understanding, early researchers train backbone networks [, , , , , ] using visual classification tasks on large-scale labeled datasets like ImageNet [] and Kinetics []. However, the high cost associated with labeled data promotes the development of self-supervised learning methods that capitalize on unlabeled data for visual pre-training [, , , , ]. Furthermore, researchers delve into multimodal pre-training utilizing large-scale weakly-supervised visual-text pairs [, , , , ], thereby enhancing their models’ capabilities and demonstrating impressive zero-shot performance. Overall, fueled by the accumulation of high-quality image and video data and advancements in visual pre-training paradigms, Video Foundation Models (VFMs) witness remarkable progress in recent years. A new generation of VFMs [, , , , , , ] emerges, demonstrating outstanding performance on conventional video understanding benchmarks.

The rapid development of VFMs raises a new problem: How to evaluate a video foundation model? In image realm, previous works assess the generalization capability of Image Foundation Models (IFMs) by evaluating their performance on numerous downstream visual tasks, encompassing diverse scenarios and evaluation protocols [, , , , , , ]. In contrast, previous works primarily evaluates VFMs through benchmarks simply focusing on action recognition tasks [, , ]. Some studies [, , ] have also considered combining VFMs with Large Language Models (LLMs) to evaluate their performance on multimodal video understanding tasks. There are several issues with current evaluation benchmarks: (1) Limited domain and task: benchmarks like Kinetics [], Something-Something [] and AVA [], which focus on action recognition, overlook other video understanding scenarios (e.g., video quality assessment), limiting their applicability in evaluating the generalization capabilities of visual foundation backbones across diverse video understanding applications. (2) Saturated performance: the performance of VFMs on the standard benchmarks [] has reached a saturation point ( around 90% Top-1 accuracy), making it challenging to differentiate between the true capabilities of different VFMs. (3) High evaluation cost: the conventional validation protocols often necessitate end-to-end training on the entire dataset, resulting in a significant challenge for large VFMs. (4) Less indicative: Incorporating LLM into VFMs for evaluation on multi-modal video tasks may introduce bias and is less indicative for vision representation power, as performance differences might stem from the language model rather than the VFMs itself.

To tackle these problems, we build a comprehensive vision-centric benchmark suite for evaluating VFMs, namely VideoEval. As shown in Figure 1, our method has the following key features: Comprehensive: First, we create the Video Task Adaptation Benchmark (VidTAB) to evaluate the adaptability of VFMs to unseen tasks with limited training samples. We collect public datasets from various video task domains, ranging from action recognition in special scenarios, AI for science, video content moderation, video quality/aesthetic assess, to emotion analysis. From these domains, we construct eight adaptation tasks and propose evaluation protocols and adaptation methods suitable for current VFMs. Additionally, to assess the capability of VFMs as a direct feature embedding extractor for downstream applications, we create the Video Embedding Benchmark (VidEB), which includes four tasks to evaluate their embedding power at different granularities.
Challenging & Indicative: Due to the diversity of test data and the more challenging evaluation protocols, our VideoEval can effectively distinguish between various VFMs that perform similarly on traditional benchmarks, providing deeper insights and more accurate evaluation on their true capabilities.
Low-cost: Thanks to our training-light few-shot evaluation and training-free feature embedding evaluation protocols, VideoEval requires significantly fewer evaluation cost compared to previous benchmarks, while still maintaining a comparable number of testing samples to ensure accurate and stable evaluations.
Vision-centric: Our evaluation focuses solely on the capabilities of video foundation models themselves, avoiding the bias due to the introduction of LLMs.

Based on VideoEval, we evaluate 20 open-source vision foundation models, including Video Foundation Models (VFMs), Image Foundation Models (IFMs), and IFMs with image-to-video adaptors. Our main findings as following: First, current VFMs still struggle to adapt to unseen video tasks with limited training samples. Second, while more data and larger models generally improve performance, augmenting video training data can sometimes negatively affect certain tasks. Third, the effectiveness of certain pre-training paradigms, such as VideoMAEv2 [], may not have been adequately validated in previous benchmarks. Finally, combining multiple pre-training paradigms can lead to models with better generalization capabilities, such as performing multimodal contrastive learning after unimodal visual self-supervised pre-training [, ].

**Table 1: Comparison of VFMs Benchmark. "Num. training" denotes number of training samples, "Num. test" denotes number of test samples, and "Beyond Action" denotes the tasks in this benchmark extend beyond action understanding.**
| Benchmark | Num. training | Num. test | Beyond Action | Task Diversity | Domain Diversity | VFMs-specific protocol |
| --- | --- | --- | --- | --- | --- | --- |
| Single-dataset Benchmarks |  |  |  |  |  |  |
| Kinetics-400 [] | 240,436 | 19,165 | ✗ | ✗ | ✗ | ✗ |
| Sth-Sth V2 [] | 168,913 | 24,777 | ✗ | ✗ | ✗ | ✗ |
| Moment-in-Time [] | 791,246 | 33,898 | ✗ | ✗ | ✗ | ✗ |
| UCF101 [] | 9,537 | 3,783 | ✗ | ✗ | ✗ | ✗ |
| Multi-dataset Benchmarks |  |  |  |  |  |  |
| SEVERE [] | 868,446 | 144,830 | ✗ | ✓ | ✓ | ✗ |
| BEAR [] | 240,236 | 140,436 | ✗ | ✓ | ✓ | ✗ |
| VideoGLUE [] | 1,896,621 | 239,011 | ✗ | ✓ | ✓ | ✓ |
| VideoEval | 5,704 | 20,497 | ✓ | ✓ | ✓ | ✓ |

## 2 Related Work

#### Video foundation models (VFMs).

With the continuous growth of image [, , ] and video data [, , , , ] and advancements in pre-training paradigms, research on Video Foundation Models (VFMs) has progressed rapidly. Current VFMs are primarily built with different pre-training paradigms: contrastive or predictive self-supervised learning methods [, , , , ] on unimodal video data, masked video modeling based on unimodal video data [, , , , , , ] and video-text contrastive learning based on multimodal visual-text pairs [, , , , , ]. Some works [, , , ] combine these paradigms, enabling VFMs to extend further into multimodal understanding. Additionally, some studies introduce modalities like audio and speech on top of video and text [, , ], further expanding the capabilities of VFMs. Recently, InternVideo2 [] combines pre-training paradigms and large-scale high-quality data to scale VFMs to 6 billion parameters, achieving remarkable performance improvements.

#### Evaluation of VFMs.

Previous works primarily utilize action recognition benchmarks [, , ] to evaluate VFMs with a focus on understanding the appearnace and motion from videos. To enhance evaluation diversity, some studies explore richer domains and tasks [, , ], but they still remain limited to the action recognition tasks. The InternVideo series [, ] and VideoGLUE [] attempt to provide a more comprehensive evaluation on VFMs by reporting performance on a number of benchmarks and evaluation protocols. However, these efforts are still based on existing benchmarks and incurred high validation costs. In contrast, our VideoEval considers the characteristics and application scenarios of VFMs, offering a comprehensive and more challenging evaluation solution from the perspectives of task definition and evaluation protocols.

## 3 Building VideoEval

Figure: Figure 2: Illustration of building VideoEval. We build VideoEval through the following steps: (1) conducting task selection by considering our expected capabilities for VFMs. (2) performing data filtration from the perspectives of quality, difficulty, and diversity, (3) standardizing the task format through task construction, and (4) defining evaluation methods and targets.
Refer to caption: x2.png

We argue that a powerful video foundation model should possess two key capabilities: (1) strong task adaptation ability, i.e., the ability to adapt to diverse and new tasks with limited training samples, and (2) the capacity of general video encoder to extract feature embedding that retain and distill key information from videos, directly supporting various downstream tasks. From these two perspectives, we construct VideoEval, which includes the Video Task Adaptation Benchmark (VidTAB) and the Video Embedding Benchmark (VidEB). By collecting diverse tasks from different scenarios and employing low-cost evaluation scheme, our VideoEval can efficiently and comprehensively assess the generalization ability of VFMs in video understanding. In this section, we present our VideoEval in detail. The construction pipeline for VideoEval is illustrated in Figure 2, and the evaluation tasks we ultimately constructed are presented in Table 2.

### 3.1 Video Task Adaption Benchmark

#### Collecting diverse datasets from public source.

Previous benchmarks primarily focus on evaluating video models for action recognition, overlooking many other video understanding tasks. Therefore, we consider five different application scenarios: Action Recognition in Special Scenarios (Action), AI for Science (Science), Video Content Moderation (Safety), Video Quality Assessment (Quality), and Emotion Analysis (Emotion). We collect publicly annotated video datasets from the Internet. Subsequently, we manually filter out data with low video or annotation quality. We also consider the difficulty level of tasks to ensure that our benchmark can distinguish between models. We discard tasks that are very easy or can even be accomplished without video understanding, as well as tasks that are extremely difficult even for humans. It does not imply that we have low expectations for foundation model capabilities relative to humans; Rather, we aim to ensure meaningful and effective evaluation in the context of task adaptation, because overly challenging tasks would not provide clear distinctions between models.

**Table 2: Task details of VideoEval. All videos are collected from the public datasets for building tasks of VidTAB and VidEB.**
| Domain | Task | Source | Task Description |
| --- | --- | --- | --- |
| Video Task Adaptation Benchmark (VidTAB) |  |  |  |
| Action Recognition<br>in Special Scenarios | Action Recognition | ARID [] | Recognizing 11 distinct human actions in dark scenarios. |
| in Dark Scene | e.g. Run / Walk / Drink |  |  |
| Action Recognition | BreakFast [] | Classifying 10 types of long-duration cooking videos. |  |
| in Long Video | e.g. Milk / Tea / Sandwich |  |  |
| AI for Science | Medical Surgery | SurgicalActions160 [] | Classifying 16 surgical actions in gynecologic laparoscopy. e.g. Knotting / Suction / Injection |
| Animal Behavior | Animal<br>Kingdom [] | Classifying 12 behaviors of wild animals from diverse environmental footage. |  |
| e.g. Flying / Chirping / Preening |  |  |  |
| Video Content<br>Moderation | Fake Face | FaceForensics++ [] | Determine whether the faces in the video have been tampered with by AI technology (such as DeepFake). |
| e.g. Origin video / Video with fake face |  |  |  |
| Harmful Content | mob [] | Detecting 3 degrees of malicious content within videos. |  |
| e.g. Obscene / Indecent activity / Violent activity |  |  |  |
| Video Quality<br>Assessment | Quality Assess | DOVER [] | Evaluating videos from an aesthetic and technical perspective and categorizing them into low and high quality. |
| e.g. Low quality / High quality |  |  |  |
| Emotion Analysis | Emotion Analysis | CAER [] | Classifying 7 different human emotions in video. |
| e.g. Happy / Fear / Anger |  |  |  |
| Video Embedding Benchmark (VidEB) |  |  |  |
| Scene Understanding<br>in Temporal Contexts | Duplicate Scene<br>Retrieval | FIVR5K [] | Retrieve Duplicate Scene Videos (DSV): |
| Videos captured by the same camera and sharing at least one scene (without considering any application transformations). |  |  |  |
| Complementary Scene<br>Retrieval | FIVR5K [] | Retrieve Complementary Scene Videos (CSV): |  |
| Retrieve a portion of the same spatiotemporal segment captured from different perspectives. |  |  |  |
| Incident Scene<br>Retrieval | FIVR5K [] | Retrieving Incident Scene Videos (ISV): |  |
| The same event is close in both space and time, but there are no overlapping videos. |  |  |  |
| Copy Detection | DVSC23 [] | Detecting edited versions of the same source video. |  |
| Given a query inserted with one or more copied segments, detect the source video from the database. |  |  |  |
|  |  |  |  |

#### Constructing the adaptation tasks based on the existing annotations.

Classification tasks are straightforward and well-defined, with high classification performance often indicating effective feature learning. Therefore, they are suitable for evaluating video foundation models. We construct adaptation classification tasks based on the collected data and annotations. For datasets that originally include category labels, such as ARID [] and Animal Kingdom [], we select categories with sufficient samples to ensure evaluation accuracy and stability. We also control the final number of categories to avoid making the adaptation task overly difficult. For datasets that are not originally classification tasks, we transform the tasks into classification tasks. For example, for DOVER [], which is used for video aesthetics and technical quality assessment (a regression task), we assume that videos with quality scores in the top 40% are "high-quality videos" and those with scores in the bottom 40% are "low-quality videos", thus converting the original task into a binary classification task. In total, we construct eight classification tasks to evaluate the adaptation capabilities of video foundation models. Specific construction details of each task are provided in the Appendix.

#### Determining the evaluation protocol and metric.

Previous studies [, , ] typically fine-tune video foundation models using the entire samples of training set, and most popular benchmarks have large training sample sizes. We argue that this evaluation method overlooks the examination of the adaptation capability of VFMs under few-shot setting. As illustrated in Figure 3, under the scenario of using full training samples, the differences between VFMs are difficult to discern. However, under a low-sample protocol, different foundation models exhibit varying degrees of task adaptation capabilities. We observe that for the task of Action Recognition in Dark Scenes, which VFMs usually excel at, there are significant differences in adaptation capabilities among different models when training samples are extremely limited (4 shot and 16 shot). As the number of samples gradually increases to 100 shot, these differences diminish. Conversely, for more challenging tasks like Emotion Analysis, the performances of different models are uniformly weak when training samples are extremely limited, showing no discernible differences until a certain number of training samples (100 shot) are reached, at which point different models begin to demonstrate distinct adaptation capabilities. Therefore, to account for the adaptation capabilities of models with different numbers of training samples, we define a new task adaptation capability evaluation score (TA-score):

$$ $\text{TA-score}=\frac{{Acc}^{4s}+Acc^{16s}+Acc^{100s}}{3},$ (1) $$

where $Acc^{4s}$, $Acc^{16s}$, $Acc^{100s}$ represent the model’s top-1 accuracy for 4-shot, 16-shot, and 100-shot classifications, respectively. Unless otherwise specified, we will use the TA-score as the metric to evaluate the performance of various tasks in VidTAB.

Figure: Figure 3: Performance comparison on different training data scales. We evaluate the performance variation of multiple video foundation models across tasks from two different domains as the scale of the training data changed. ’FT’ and ’AP’ denote full finetuning and attentive probe, respectively.
Refer to caption: x3.png

Figure: Figure 4: Illustration of different adaptation method: (a) Full Finetuning, (b) Adapter, (c) Attentive Probe, and (d) Linear Probe.
Refer to caption: x4.png

**Table 3: Comparison of adaptation method on V-JEPA-H [] All results are obtained using A100-80G with PyTorch-builtin mixed precision, using a batch size of 4 to measure CUDA memory and training time. "Dark" and "Emotion" denote the tasks of Action Recognition in Dark Scenes and Emotion Analysis, respectively.**
| Adaptation<br>method | Tunable<br>Params (M) | CUDA<br>Memory (G) | Training<br>Time (h) | Dark<br>TA-score | Emotion<br>TA-score |
| --- | --- | --- | --- | --- | --- |
| full finetuning | 663.7 | 52.1 | 1.0 | 68.8 | 25.3 |
| adapter | 52.6 | 45.0 | 1.0 | 62.4 | 24.7 |
| attentive probe | 19.7 | 6.4 | 0.4 | 54.7 | 23.8 |
| linear probe | 0.0 | 6.0 | 0.3 | 12.9 | 16.2 |
|  |  |  |  |  |  |

#### Identifying efficient adaptation method for evaluation.

We also need to investigate how to adapt the video foundation models to the downstream tasks. Previous work [, , , , , , ] has explored various strategies for efficient adapting the foundation models. Here, we consider several popular methods, as illustrated in Figure 4: Full Finetuning: Fine-tuning all the parameters of the pre-trained model. Adapter: Freezing the pre-trained model and inserting learnable low-rank adapter [] modules into each block of the pre-trained model for adaptation. Attentive Probe: Freezing the pre-trained model and adding an additional learnable cross-attention block at the end of the model to achieve attentive pooling, followed by a linear projection for classification. Linear Probe: Directly using the features from the pre-trained model, performing mean pooling, and then using a linear projection for classification. We evaluate the performance of these adaptation methods based on the V-JEPA-H model, as shown in Table 3. Full finetuning and adapter achieve the best adaptation performance, but require high training costs. Linear probe is highly efficient but showed weak adaptation performance. Attentive probe offers a good trade-off between efficiency and adaptation performance. Therefore, in subsequent evaluation experiments, we employ attentive probe to adapt various video foundation models.

### 3.2 Video Embedding Benchmark

Apart from the traditional classification tasks, the more direct evaluation of representations typically involves the retrieval tasks such as video action retrieval [, , ], which primarily rely on text information. However, these benchmarks often overlook the overall scene context and exhibit an overlap with recognition tasks. In contrast, inspired by previous works [, , , , ], we focus on vision-centric embedding evaluation in Table 2. Specifically, we require models to rank and recall instance-level samples based on similarity, rather than solely on action categories. This evaluation protocol focuses on the model’s capability of capturing the subtle visual information.

Evaluation protocol. To facilitate fine-grained embedding evaluation, we include two tasks for assessment: (1) Hierarchical Video Retrieval aims to retrieve videos from a database that closely matches the query video in terms of scene, viewpoint, and temporal context. According to previous work [], videos related to the query are categorized into three levels based on their similarity to the query: Duplicate Scene Videos (DSVs), Complementary Scene Videos (CSVs), and Incident Scene Videos (ISVs), as shown in Table 2.
Consequently, the retrieval tasks are structured into three hierarchical levels:

- •
Duplicate Scene Video Retrieval: only DSVs are positive instances.
- •
Complementary Scene Video Retrieval: both DSVs and CSVs are positive instances.
- •
Incident Scene Video Retrieval: DSVs, CSVs, and ISVs are all positive instances.

For the evaluation metric, we follow [] to utilize the mean Average Precision (mAP) to assess the quality of video ranking.
(2) Video Copy Detection aims to detect edited copies of the query video. In contrast to the ranking/retrieval task where all video pairs need to be sorted according to video embedding similarity, it is required to identify a set of video pairs that contain edited versions of the given query. Following [], we consider the micro-AP ($\mu$AP) as our evaluation metric that operates on all queries jointly and takes the confidence scores
into account.

## 4 Benchmarking Video Foundation Models

### 4.1 Video foundation models and evaluation details

#### Evaluation targets.

We evaluate twenty open-source vision foundation models. Including: (1) twelve video foundation models, covering different pre-training paradigms, model scales, and training data scales, to analyze the impact of these factors on the generalization capability of foundation models. (2) five image foundation models to observe how much generalization capability trained on image data can exhibit in video understanding. (3) three image-to-video methods based on image foundation models to assess the effectiveness of current efficient transfer methods.

#### Implementation details.

All models take 8 frames (16 frames if the model has temporal downsampling), with each frame being 224x224 in size as input. For VidTAB, to ensure fair comparison and efficient assessment, we train all models for the same number of epochs and make minor adjustments to the hyperparameters to ensure convergence.
For VidEB, all models take 16 frames, with each frame of size 224x224.
In hierarchical video retrieval, the similarity of video-level embedding determines the ranking of retrieval results.
In video copy detection, each sample is segmented into 5 clips. The detection confidence score for the entire video is derived from the maximum frame-wise similarity computed for each query-reference pair.
See the appendix for more details.

### 4.2 Results on VidTAB

**Table 4: Evaluating state-of-the-art FMs on the VidTAB. The best and second-best results of foundation models are noted by blue and underline, respectively. ’I’, ’V’, and ’IV’ denote image data, video data, and mixed image-video data, respectively. Data marked in gray indicates that the model uses a model trained on that data as initialization. ’K710ft’ indicates that the model was fine-tuned with supervision using the labeled action recognition dataset Kinetics-710 (0.66M).**
|  |  |  |  | Action | Science | Safety | Quality | Emotion |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| # Params (M) | # Pretrain Data | Average | Dark Scene | Long Video | Medical Surgery | Animal Behavior | Harmful Content | Fake Face | Quality Assess | Emotion Analysis |  |
| Random | - | - | 22.7 | 9.1 | 10.0 | 6.3 | 8.3 | 33.3 | 50.0 | 50.0 | 14.3 |
| Zero-shot performance of visual language models |  |  |  |  |  |  |  |  |  |  |  |
| CLIP-L [] | 428 | I-400M | 35.7 | 29.2 | 34.6 | 12.5 | 32.9 | 42.1 | 56.3 | 65.5 | 12.9 |
| EVA-CLIP-g [] | 1365 | I-2B | 36.0 | 32.8 | 37.2 | 9.4 | 28.5 | 39.6 | 52.8 | 69.5 | 17.9 |
| ViCLIP-L [] | 428 | I-400M+V-200M | 33.6 | 26.2 | 37.5 | 8.3 | 29.3 | 32.1 | 52.2 | 53.9 | 29.0 |
| InternVideo2stage2 [] | 1350 | IV-1.1M+IV-25.5M | 40.6 | 37.1 | 40.2 | 11.5 | 45.2 | 59.1 | 51.3 | 56.1 | 24.3 |
| Image Foundation Model |  |  |  |  |  |  |  |  |  |  |  |
| CLIP-L [] | 316 | I-400M | 42.8 | 31.2 | 38.0 | 32.3 | 36.3 | 50.3 | 58.5 | 67.7 | 28.1 |
| SigLiP-SO [] | 444 | I-4.11B | 42.6 | 26.9 | 37.6 | 36.5 | 35.7 | 49.4 | 58.2 | 68.1 | 28.2 |
| EVA-g [] | 1035 | I-2B | 45.4 | 39.7 | 45.9 | 34.4 | 40.0 | 50.8 | 55.2 | 68.5 | 28.6 |
| DINOv2-L [] | 317 | I-142M | 40.7 | 37.9 | 41.0 | 39.6 | 35.7 | 34.3 | 52.0 | 62.2 | 23.2 |
| DINOv2-g [] | 1165 | I-142M | 43.0 | 35.8 | 43.5 | 42.7 | 35.6 | 44.0 | 53.2 | 63.7 | 25.3 |
| Image Foundation Model with image-to-video adaptation method |  |  |  |  |  |  |  |  |  |  |  |
| ST-Adapter-CLIP-L [] | 328 | I-400M | 46.9 | 43.0 | 45.0 | 31.2 | 39.4 | 49.4 | 64.9 | 72.3 | 29.9 |
| AIM-CLIP-L [] | 328 | I-400M | 48.7 | 42.3 | 49.9 | 38.5 | 38.9 | 47.0 | 68.3 | 74.6 | 30.1 |
| ZeroI2V-CLIP-L [] | 303 | I-400M | 46.5 | 41.3 | 46.8 | 31.2 | 39.3 | 47.2 | 64.6 | 70.6 | 30.6 |
| Video Foundation Model |  |  |  |  |  |  |  |  |  |  |  |
| ViCLIP-L-10M [] | 316 | I-400M+V-10M | 41.4 | 31.3 | 41.2 | 30.2 | 34.2 | 45.3 | 53.9 | 68.3 | 26.8 |
| ViCLIP-L-200M [] | 316 | I-400M+V-200M | 42.7 | 36.7 | 43.9 | 30.2 | 36.8 | 46.9 | 54.8 | 65.4 | 27.2 |
| VideoMAEv1-H [] | 651 | V-0.24M | 44.4 | 45.1 | 30.3 | 35.4 | 37.9 | 53.9 | 52.0 | 71.3 | 28.9 |
| VideoMAEv2-g [] | 1037 | V-1.35M | 37.7 | 34.9 | 18.5 | 18.8 | 33.3 | 57.6 | 50.7 | 67.6 | 20.4 |
| VideoMAEv2-g [] | 1037 | V-1.35M+K710ft | 53.5 | 76.1 | 72.4 | 50.0 | 42.6 | 43.3 | 56.6 | 61.5 | 25.1 |
| UMT-Lstage1 [] | 316 | V-0.66M | 40.0 | 33.9 | 34.4 | 30.0 | 33.5 | 44.1 | 53.4 | 64.9 | 26.1 |
| UMT-Lstage2 [] | 316 | V-0.66M+IV-25M | 43.6 | 32.6 | 44.0 | 22.9 | 39.3 | 62.3 | 52.8 | 67.7 | 27.2 |
| V-JEPA-L [] | 318 | V-2M | 43.0 | 50.2 | 34.1 | 39.6 | 38.5 | 41.9 | 51.5 | 66.9 | 21.0 |
| V-JEPA-H [] | 653 | V-2M | 44.5 | 54.7 | 36.0 | 35.4 | 40.5 | 45.1 | 52.9 | 67.7 | 23.8 |
| InternVideo2-1Bstage1 [] | 1037 | IV-1.1M | 45.1 | 45.4 | 47.7 | 33.3 | 37.7 | 48.9 | 53.7 | 66.0 | 27.8 |
| InternVideo2-1Bstage1 [] | 1037 | IV-1.1M+K710ft | 56.4 | 75.9 | 77.9 | 53.1 | 45.2 | 47.7 | 55.6 | 64.7 | 30.9 |
| InternVideo2-1Bstage2 [] | 1037 | IV-1.1M+IV-25.5M | 53.1 | 67.6 | 69.5 | 38.5 | 48.6 | 51.2 | 55.2 | 64.5 | 29.4 |
|  |  |  |  |  |  |  |  |  |  |  |  |

#### Zero-shot evaluation.

To preliminarily test the difficulty of the dataset, we first evaluate the zero-shot performance of the eight tasks using two image language models and two video language models. As shown in the top section of Table 3, both image and video models demonstrate relatively high performance for action-related tasks, and video models achieve better. For tasks involving low-level information understanding, such as Quality Assessment task, image models perform significantly better. In contrast, for other tasks involving scenarios typically unseen in training data, such as medical surgery videos or Safety Review tasks requiring complex semantic reasoning, all models perform similar to random guess.

#### Main results of adapting foundation models.

Table 4 presents the evaluation results on VidTAB. We summarize our findings as follows.

In general, (1) Despite exhibiting a degree of generalization capability, current vision FMs still struggle to adapt to unseen video tasks with limited training samples. VFMs outperform IFMs, particularly in tasks related to action and behavior understanding. However, IFMs achieve superior performance on more low-level tasks, particularly in the domains of safety and quality, especially when combined with image-to-video adaptation techniques.

(2) The adaptation performance of models generally increases with the growth of data and model size, as observed by the improvements observed from V-JEPA-L to V-JEPA-H (+1.5) and ViCLIP-L-10M to ViCLIP-L-200M (+1.3).

For the training data, (3) While augmenting video training data is generally beneficial, it can negatively impact the performance on some tasks. For both VideoMAEv2-g and InternVideo2-1Bstage1, fine-tuning on Kinetics-710 data significantly enhances Action-related tasks, but consistently degrades the performance on the Safety and Quality tasks. Similar findings are observed with ViCLIP-L, where post-pretraining on a large-scale video dataset improves Action-related tasks but diminishes performance in other domains (Science, Safety, Quality, Emotion). It could be attributed to the limited diversity of the current video training data.

(4) For models trained on single-modal visual data, incorporating additional weak-supervised post-pretraining with visual-text data leads to significant improvements in adaptation capabilities. This is evident in the performance gains observed from UMT-Lstage1 to UMT-Lstage2 (+3.6) and from InternVideo2-1Bstage1 to InternVideo2-1Bstage2 (+8.0). Interestingly, this finding contradicts previous conclusions drawn from commonly used action recognition benchmarks, suggesting that these benchmarks may introduce bias.

For the pre-training paradigms of model, (5) The effectiveness of pre-training paradigms in scaling model size might not be adequately validated on popular action recognition benchmarks [, ]. While VideoMAEv2 [] successfully scaled a model to 1B parameters using the dual masking strategy, its adaptation performance (37.7 vs 44.4) significantly declined compared to VideoMAEv1-H. Interestingly, VideoMAEv2-g demonstrated remarkable performance after fine-tuning on Kinetics-710 (0.66M), suggesting that the abundant labeled data may have compensated for the shortcomings of its pre-training performance.

(6) Single-modal self-supervised pre-training paradigms exhibit superior data efficiency compared to multimodal weakly-supervised pre-training paradigms. Notably, V-JEPA and VideoMAEv1, trained solely on relatively small-scale unlabeled video data via self-supervised pre-training, demonstrate comparable or even superior performance to ViCLIP, which is trained on a massive dataset of video-text pairs.

Finally, (7) Effective adaptation method for image-to-video FMs is crucial. Three image-to-video methods based on CLIP-L achieved significant performance improvements compared to using an attentive probe directly. We believe this represents a promising avenue for future research.

**Table 5: Evaluation of State-of-the-Art Foundation Models on the VidEB Dataset. "K400pt" and "K400ft" denote that the model is pre-trained and fine-tuned, respectively, using the labeled action recognition dataset Kinetics-400 (0.31M). MCL: Multi-modal Contrastive Learning, SCL: Self-supervised Contrastive Learning, MVM: Masked Video Modeling, SFT: Supervised Fine-tuning. Other notations are consistent with those in Table 4.**
|  |  |  |  | Scene |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Pretrain Tasks | # Pretrain Data | Average | Duplicate | Complementary | Incident | Copyright |  |
| Image Foundation Model |  |  |  |  |  |  |  |
| CLIP-L [] | MCL | I-400M | 43.0 | 41.1 | 46.4 | 52.0 | 32.3 |
| EVA-g [] | MCL | I-2B | 37.1 | 41.4 | 46.1 | 51.7 | 9.3 |
| SigLiP-SO [] | MCL | I-4.11B | 38.6 | 40.6 | 45.5 | 51.5 | 16.9 |
| DINOv2-L [] | SCL | I-142M | 45.6 | 49.0 | 53.5 | 54.3 | 25.6 |
| DINOv2-g [] | SCL | I-142M | 48.6 | 50.5 | 55.1 | 56.0 | 32.8 |
| Video Foundation Model |  |  |  |  |  |  |  |
| ViCLIP-L-10M [] | MVM+MCL | I-400M+V-10M | 43.7 | 40.0 | 44.4 | 46.9 | 43.4 |
| ViCLIP-L-200M [] | MVM+MCL | I-400M+V-200M | 39.9 | 33.1 | 37.2 | 40.3 | 49.2 |
| VideoMAEv1-L [] | MVM | K400pt | 12.9 | 14.5 | 15.1 | 13.2 | 8.8 |
| VideoMAEv1-L-K400ft [] | MVM+SFT | K400pt+ft | 27.4 | 27.6 | 30.2 | 30.3 | 21.6 |
| VideoMAEv2-g [] | MVM | V-1.35M | 11.6 | 14.8 | 15.4 | 13.4 | 2.8 |
| VideoMAEv2-g-K710ft [] | MVM+SFT | V-1.35M+K710ft | 37.4 | 33.8 | 37.1 | 37.1 | 41.7 |
| UMT-Lstage1 [] | MVM | V-0.66M | 41.1 | 42.2 | 46.6 | 49.6 | 25.7 |
| UMT-Lstage1-K710ft [] | MVM+SFT | V-0.66M+K710ft | 29.0 | 26.4 | 29.4 | 30.3 | 30.0 |
| UMT-Lstage2 [] | MVM+MCL | V-0.66M+IV-25M | 34.2 | 33.4 | 37.3 | 40.6 | 25.4 |
| V-JEPA-L [] | MVM | V-2M | 19.7 | 21.3 | 23.9 | 21.7 | 12.0 |
| V-JEPA-H [] | MVM | V-2M | 20.2 | 21.5 | 23.7 | 21.2 | 14.3 |
| InternVideo2-1Bstage1 [] | MVM | IV-1.1M | 50.4 | 47.3 | 52.1 | 54.9 | 47.3 |
| InternVideo2-1Bstage1-K710ft [] | MVM+SFT | IV-1.1M+K710ft | 33.9 | 30.5 | 34.2 | 34.1 | 36.9 |
| InternVideo2-1Bstage2 [] | MVM+MCL | IV-1.1M+IV-25.5M | 34.6 | 32.4 | 36.8 | 39.9 | 29.3 |
|  |  |  |  |  |  |  |  |

### 4.3 Results on VidEB

The main results of VidEB are presented in Table 5. We evaluate the embedding performance using different pre-training paradigms for IFMs and VFMs as frozen feature extractors. Surprisingly, IFMs performs better than most VFMs, likely due to the existing gap in spatial modeling capabilities between VFMs and IFMs.

For the pre-training paradigms of the model,
(1) The contrastive learning (CL) based approach consistently excels in embedding evaluation.
Due to CL’s emphasis on the relationships between samples during training, DINOv2, which focuses solely on vision, outperforms vision-language contrastive methods like CLIP across multiple tasks.

(2) The effectiveness of masked video modeling is closely tied to the targets it reconstructs or aligns with.
With higher semantic richness, it shows progressive improvements in embedding quality for VideoMAE-L, V-JEPA-L, and UMT-Lstage1.

(3) Vision-centric pretraining outperforms multi-modal pretraining in vision-centric scenarios.
Comparing UMT-Lstage1 and InternVideo2-1Bstage1 with their multi-modal counterparts UMT-Lstage2 and InternVideo2-1Bstage2, the introduction of visual-text pair data in multi-stage training does not enhance performance in vision-centric scenarios. This is also consistent with the performance differences observed between DINO and CLIP-style pre-training methods.

Additionally, we assess the impact of fine-tuning on the embedding evaluation of these pre-trained models. (4) Labels bring new semantic information or disrupt existing finer-grained semantic information. The performance variations after fine-tuning differ based on the pre-training strategy. For UMT-Lstage1 and InternVideo2-1Bstage1, fine-tuning leads to a significant drop in performance (-12.1 for UMT and -16.5 for InternVideo) due to the introduction of more singular label information, which causes catastrophic forgetting. In contrast, VideoMAE and VideoMAEv2 show substantial performance gains (+14.5 and +25.8, respectively) because the low-level semantics learned during pre-training are less abstract and benefit more from the addition of high-level label information.

## 5 Conclusions

We have presented the VideoEval, a comprehensive benchmark suite for efficiently evaluating the VFMs. To this end, we first establish the VidTAB benchmark, which explores suitable evaluation tasks and protocols for VFMs from the perspective of assessing their adaptability to unknown tasks with limited samples. Additionally, we create the VidEB benchmark to evaluate the capability of VFMs as direct feature embedding extractors for the downstream tasks. Based on the VideoEval, we conduct a large-scale study involving 20 popular open-source vision foundation models, providing valuable insights for future research directions.

## Appendix A Appendix

In this appendix, we provide more details of VideoEval from the following aspects:

- •
Details of our benchmark are in § A.1.
- •
Details of training and evaluation, can be found in § A.2.
- •
Ethics etatement of the datasets are in § A.3
- •
Limitations and potential negative societal impacts are in § A.4

### A.1 Details of Benchmark

Figure: Figure 5: Examples of VidTAB. We present video examples for each task in VidTAB, demonstrating that successfully completing these tasks requires VFMs to possess strong generalization capabilities.
Refer to caption: x5.png

#### Examples of VidTAB.

As shown in Figure 5, we present some examples of tasks in VidTAB.

**Table 6: Task construction details of VidTAB. All videos are collected from the public datasets for building tasks of VidTAB.**
| Task | Source | Num. test sample | Details of Task Construction |
| --- | --- | --- | --- |
| Action Recognition | ARID [] | 2011 | We directly employ the original classification task definition. Specifically, 11 categories. |
| Action Recognition | BreakFast [] | 822 | We directly employ the original classification task definition. Specifically, 12 categories. |
| Medical Surgery | SurgicalActions160 [] | 96 | We directly employ the original classification task definition. Specifically, 16 categories. |
| Animal Behavior | Animal Kingdom [] | 2268 | Since the annotations in this dataset included multiple labels, we filtered out all categories with only single labels and then |
| selected categories with more than 150 samples. This resulted in a final set of 12 categories. |  |  |  |
| Fake Face | FaceForensics++ [] | 1800 | We used the original 1000 videos as positive samples. Then, we divided the original videos into five parts and used the Deepfakes, |
| Face2Face, FaceShifter, FaceSwap, and NeuralTextures methods to generate 1000 negative samples by face-swapping. We then |  |  |  |
| selected 1800 of these samples as the test set and the remaining as the training set. |  |  |  |
| Harmful Content | mob [] | 1661 | We categorized videos into three classes based on their content: those containing fast repetitive movements and violence activities, |
| those containing unpleasant appearances and obscene scenes, and those containing no malicious information at all. This resulted in a |  |  |  |
| three-class classification task. |  |  |  |
| Quality Assess | DOVER [] | 724 | To convert the task into a classification problem, we sorted the "overall score" label and divided the videos into positive and negative |
| samples, with the top and bottom 40% constituting the respective categories. |  |  |  |
| Emotion Analysis | CAER [] | 3953 | We directly employ the original classification task definition. Specifically, 7 categories. |
|  |  |  |  |

#### Details of VidTAB.

The detals of task construction are presented in Table 6. For each category in one task, we sample 4, 16, and 100 samples, respectively. Given the limited volume of medical surgery data, we only sample 4 samples from each category for few-shot evaluation. To mitigate the impact of randomness, we sampled two sets of data for four tasks and obtained the benchmark results. We found that the randomness of sampling had negligible effects on the final rankings of VFMs in the benchmark.

### A.2 Details of Training and Evaluation

#### Code and annotations

We release the evaluation code and the modified annotation file on [https://github.com/leexinhao/VideoEval](https://github.com/leexinhao/VideoEval), see README.md for details.

#### Checkpoints of Evaluation Models

We provide checkpoints of the models we evaluate for reproducibility of our results.

- •
CLIP []: [https://huggingface.co/openai/clip-vit-large-patch14](https://huggingface.co/openai/clip-vit-large-patch14)
- •
EVA-CLIP []: [https://huggingface.co/QuanSun/EVA-CLIP](https://huggingface.co/QuanSun/EVA-CLIP)
- •
ViCLIP []: [https://github.com/OpenGVLab/InternVideo/tree/main/Data/InternVid](https://github.com/OpenGVLab/InternVideo/tree/main/Data/InternVid)
- •
InternVideo2 []: [https://huggingface.co/collections/OpenGVLab/internvideo2-6618ccb574bd2f91410df5cd](https://huggingface.co/collections/OpenGVLab/internvideo2-6618ccb574bd2f91410df5cd)
- •
SigLiP []: [https://huggingface.co/google/siglip-so400m-patch14-384](https://huggingface.co/google/siglip-so400m-patch14-384)
- •
DINOv2 []: [https://huggingface.co/facebook/dinov2-giant](https://huggingface.co/facebook/dinov2-giant)
- •
VideoMAE []: [https://github.com/MCG-NJU/VideoMAE/blob/main/MODEL_ZOO.md](https://github.com/MCG-NJU/VideoMAE/blob/main/MODEL_ZOO.md)
- •
VideoMAEv2 []: [https://github.com/OpenGVLab/VideoMAEv2/blob/master/docs/MODEL_ZOO.md](https://github.com/OpenGVLab/VideoMAEv2/blob/master/docs/MODEL_ZOO.md)
- •
UMT []: [https://github.com/OpenGVLab/unmasked_teacher](https://github.com/OpenGVLab/unmasked_teacher)
- •
V-JEPA []: [https://github.com/facebookresearch/jepa](https://github.com/facebookresearch/jepa)

#### Trainging strategies.

Specific hyperparameter configurations are available in the configs provided in our code repository. In essence, we train all models for 25 epochs using a similar training strategy, employing the Adam optimizer, a learning rate of 5e-5, and only utilizing RandomResizedCrop for data augmentation. And we use a single clip to obtain the final evaluation performance.

#### Total amount of compute and the type of resources used.

Leveraging the low cost of our evaluation protocol, we conducted each experiment involving a single VFM and a single task on one A100-80G GPU. We performed approximately 300 such experiments, each taking around 1-2 hours, resulting in a total of around 400 GPU hours.

### A.3 Ethics Statement

#### License of the datasets

The dataset we are using is collected from publicly accessible sources, all licensed under Creative Commons (CC-BY) or other open-source licenses. We have diligently followed all legal requirements to integrate this data into our research, emphasizing the importance of transparency in data licensing for proper attribution and appropriate use. Although we have taken steps to ensure the inclusion of suitable content, we recognize that some problematic content may still exist. If you encounter any such content, please notify us immediately so we can take corrective action to maintain a dataset free from inappropriate material. We are dedicated to maintaining a high-quality, ethically responsible dataset and pledge to uphold principles of privacy and transparency in all our work.

#### Privacy or safety concerns in video

For personally identifiable information or offensive content in video, our data collection sources have been carefully considered, and we believe these issues are not present. However, if you discover any oversights, please do not hesitate to contact us promptly.

### A.4 Limtiations and Societal Impacts

#### Limitations.

First, due to the limitations of diversity and accuracy in our video sources and annotations, which were gathered from public resources, we plan to further enrich the task in the future by incorporating manual annotations and self-collected data. Secondly, considering the evaluation cost and simplicity, we currently only evaluate tasks like classification and retrieval, which primarily rely on VFMs’ global information extraction capabilities. We have not yet considered tasks like spatio-temporal action detection and temporal grounding, which assess other aspects of VFMs’ capabilities. We will expand the scope of evaluation in the future.

#### Potential negative societal impacts.

While our evaluation includes tasks like synthetic video recognition and harmful information recognition, these serve only as indicators of the model’s overall performance in this area and cannot be used to accurately evaluate the actual performance of a specific task. If researchers or engineers in society attempt to use VFMs to perform these specific tasks, our benchmark can serve as a reference for their choice of VFMs, but it cannot be used as the final standard for evaluating that task. Otherwise, it may have negative impacts on the corresponding real-world applications.

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: * L482 "For VidTAB, to ensure fair comparison and efficient assessment, we train all models for the same number of epochs and made minor adjustments to the hyperparameters to ensure convergence." I am not convinced that minor adjustments to the hyperparameters would lead to a fair comparison to the model performance, especially on the few-shot setup. It would be valuable to provide the hyperparameters tuning details and make sure all models reach the best performance, for a fair comparison and conclusion.
soundness: 3
strengths: * This paper broadens the scope of existing video benchmark suites by integrating a wider range of tasks, contributing to the development of a more comprehensive evaluation framework.  
* The authors' comprehensive evaluation of numerous publicly available vision foundation models is commendable. The resulting data serves as a valuable reference and facilitates analysis of the current research landscape.
* This work enhances existing video benchmark suites by incorporating evaluation cost as a key factor, offering significant practical advantages.
summary: The paper proposes an ensemble of video datasets for foundation model benchmarking, and provides different evaluation protocols. The proposed benchmarks consist of two application scenario, namely VidTAB for few-shot classification and VidEB for zero-shot retrieval. The paper evaluated a number of publicly available image and video foundation models and provides some observations based on the benchmark results. 

Pros: 
* The paper propose to extend the evaluation datasets and evaluation methods based on existing works.
* The paper adds more publicly available foundation models to benchmark.

Cons:
* Some selection of datasets and evaluation protocols are not clearly justified.
* Some claims made by the paper maybe not be well upheld by the experiments.

Please see the below session for details.
weaknesses: * Justification of dataset selection 1: In VidTAB, why "Action Recognition in Special Scenarios" are focused instead of accessing general action recognition? Furthermore, it is not clear to me why "dark scene" is representative in action recognition in special scenarios. More justifications like reference to related works are needed.
* Justification of dataset selection 2: I have concerns regarding the validity and rationale behind the dataset choices in VidEB, particularly the focus on scene retrieval tasks. The superior performance of image-based foundation models on these benchmarks suggests a potential bias towards static scene understanding, which may not fully capture the desired motion and dynamic understanding capabilities expected in a video context.
* Following data selection 2: as the evaluation sets are towards static scene understanding, I found it hard to ground the observations and conclusion in VidEB section on video foundation models. Maybe one way to improve the task is to re-purpose some video-centric dataset for embedding retrieval task, e.g. using nearest neighbor retrieval for zero-shot classfication.
* Model sizes in the comparison in the paper: in the main table (Tab 5, 6), foundation models of different sizes are put together to compare. As it largely follows the intuition that the larger the model sizes the better the performance, I found it hard to get insights when comparing models of different size in the table. 
* It is strange to see the "zero-shot performance of visual languange models" in table 5. With TA-score proposed in eq. 1, how the score on zero-shot V-L model performance is calculated? as I understand V-L retrieval does not involve in few-shot examples.
* [minor] Table 2 characters are too small to read.

---

confidence: 5
contribution: 1
initial_rating: 3
presentation: 3
questions: Are video-language models regarded as video foundation models?

Can the benchmark be used to evaluate video-language models such as GPT-4V, video-llava, etc?

Does the benchmark cover surveillance monitoring scenarios where there may be multiple people in the scene and each person performs a different action?

Does the benchmark cover the capability to evaluate how well a person performs certain actions?
soundness: 2
strengths: The dataset is useful for researchers to evaluate their video encoders.
summary: The paper presents a benchmark dataset for the evaluation of video foundation models. It is focused on video classification (instead of video -to-text, video-question-answering, etc.). All the source videos in the dataset are selected from existing public datasets. The paper reported results for a number of existing video foundation models.
weaknesses: The paper did not evaluate any of the large multimodality models such as GPT-4V, Gemini, video-llava, etc. It seems that the paper regards "video foundation models" as "video encoders", and the benchmark can only be used to evaluate video encoders.
 
The paper focuses on classification tasks. Modern video understanding tasks have shifted from classification to more fine-grained understanding such as (dense) text description, video question answering, etc. 

Only a few (8 vidTab, 16 vidEB) frames are selected from each video. The benchmark won't be able to evaluate long video understanding capabilities, and it won't be able to capture the speed of motion.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 4
questions: * Will this benchmark be publicly available? If so, this would be a major contribution and should be mentioned in the paper.  
* Tab. 4 motivates the choice to use attentive probe as the adapter method of choice for the evaluation by showing its good cost-performace tradeoff. However, results are only shown on V-JEPA-H. Do the findings hold true for other methods?  
* As a reader unfamiliar with the attentive probe adapter, I was not able to understand how it works from just Fig. 4\. Could authors provide literature references and maybe expand the explanation in Sec. 3.1?  
* Sec. 3.1: What other tasks / datasets were considered that did not end up in the benchmark? What were the reasons for them being excluded?  
* In Tab. 5 it would be interesting to see which models perform well at spatial vs. temporal tasks. Have you considered dividing the evaluation results into the categories from Tab. 3?  
* Fig. 3 motivates the choice of averaging three few-shot settings for scoring by showing that few-to-medium shot settings better separate the model accuracies. However this is only shown on two tasks. Do the other tasks exhibit the same separation properties?  
* Sec. 4.1: How are the 8 / 16 frames that are fed into the models sampled from the videos? Does the evaluation use multiple temporal crops?  
* Sec. 4.1: How are image models applied to video?  
* I do not understand conclusion (5) in Sec. 4.2. “The effectiveness of pre-training paradigms in scaling model size might not be adequately validated on popular action recognition benchmarks.” What benchmarks is scaling not validated on and what are the results? What does “adaptation performance” mean? Could the authors clarify the meaning of this statement and explain how the following discussion supports it?  
* Some numbers in Sec. 4.2 do not match the table, e.g. “+3.6” in l 470, 8.0 in l. 471 and 37.7, 44.4 in l. 476\.  
* Sec. 4.3: How are videos ranked when using image models?  
* l. 521: “This is also consistent with the performance differences observed between DINO and CLIP-style pre-training methods.” Which results does this statement refer to?  
* Sec. 4/3, conclusion 4: “Labels bring new semantic information or disrupt existing finer-grained semantic information.” Could authors provide more context on the pre-training data used for the models discussed in this section? The conclusions are hard to understand without this context.  
* l. 471: “previous conclusions” should be cited.  
* l. 484: “Effective adaptation method for FMs is crucial.” This conclusion is not clear to me. Was the attentive probe not used for image-to-video methods? If not, which results are they being compared to?

Minor points:

* It’s often difficult to map author names of cited approaches to the actual names of the approaches. For citations that propose an approach with a well-known name, I’d suggest providing the names along with the citation, e.g. “V-JEPA (Bardes et al., 2023)”. That would also make it easier to map the methods mentioned in the related work section to Tab. 5\.  
* Citations should have parentheses around them, otherwise it looks like they are part of the sentence. Please use “\\citep” to achieve this.  
* The name VideoEval is very generic and would apply to any video benchmark. I’d suggest choosing a more specific name, e.g. “VideoFMEval”  
* Minor grammatical error in the title: Models should be plural, so the title should be “\[...\] evaluation of video foundation model**s**”  
* The font size of Tab. 2 is a little small and the table is hard to read in print.  
* The radar chart in Fig. 1 (top right) is very interesting but hard to read at this size. I’d suggest enlarging it or even moving it into its own figure.  
* Please right-justify numeric columns in tables.  
* l. 413: The table reference should be changed to Tab. 5\.  
* Could authors better explain the terms DSVs, CSVs and ISVs in Sec. 3.2?
soundness: 2
strengths: Originality:

* The paper performs a large-scale evaluation of video foundation models on a new benchmark consisting of a diverse set of tasks. While this is not the only such survey, I am not aware of any studies that evaluate such a large amount of VFMs on such a large diversity of tasks, so I see the main novelty in the sheer breadth of the evaluation and the conclusions drawn from the state of the art in video foundation models.

Quality:

* The benchmark was carefully curated to contain high quality, challenging video datasets with good diversity.  
* The paper evaluates 20 different models, including image-only baselines, image models adopted to video and video foundation models.  
* The benchmark suite is comprehensive and diverse, consisting of 6 different domains and 12 different tasks and thus offers a good overview of VFM applicability to different tasks  
* Design / scope choices made are clearly motivated and backed up with experiments, e.g. focus on few-shot setting, using attentive probe for model adaptation.

Clarity:

* The paper is well-organized and well-written.

Significance:

* Both adaptation and retrieval with video embeddings are evaluated  
* The paper draws several interesting conclusions from their findings that point out weaknesses and directions for further research.
summary: The paper introduces a benchmark for the evaluation of video foundation models (VFM). The benchmark is carefully curated based on a selection of existing video datasets. It is split into two sets: (1) a task adaptation benchmark (Video-TAB) that tests model performance on a variety of video tasks after lightweight adapter-based fine-tuning, and (2) an embedding benchmark (VidEB) that tests the performance of video embeddings, extracted using the benchmarked foundation models, on retrieval-related video tasks. Based on these benchmarks, the paper provides a survey of 20 current VFMs and offers insights into their applicability to different tasks. Furthermore, it discusses the effect of training data and training recipes on model performance.
weaknesses: It seems to me that there is a disconnect between the discussion of the results and the actual results. Conclusions drawn are not supported by the data while some clear trends in the data are not discussed. While I would not recommend this paper for acceptance in its current state, I would be in favor of accepting it if the weaknesses above can be addressed sufficiently.

* The main weakness of the paper is that many of the conclusions it draws are not fully supported by the experimental results.  
  * On VidTAB:  
    * l. 421: “current vision FMs struggle to adapt to unseen video tasks”. What is the evidence for this statement? A comparison of VFM performance to SotA on these datasets would help support this.  
    * l. 422: “VFMs outperform IFMs”: The best IFM beats 9 out of the 12 evaluated VFMs, so this statement doesn’t seem to be generally true, and also not in action and behavior tasks where some IFMs show stronger performance than most VFMs. So this statement should be weakened and results discussed in more nuance. This is actually an interesting negative result that warrants further discussion.  
    * l. 431: “Similar findings are observed with ViCLIP-L, where post-pretraining on a large-scale video dataset improves Action-related tasks but diminishes performance in other domains (Science, Safety, Quality, Emotion)”: This statement is not supported by results in Tab. 5: VICLIP-L performance actually increases on science tasks, only slightly drops on one safety task while improving on the other, and increases on emotion.  
  * On VidEB:  
    * Conclusion (1): “The contrastive learning (CL) based approach consistently excels in embedding evaluation.”: This does not seem to be the case as two MVM methods perform better than the best CL method.  
    * Conclusion (2): “The effectiveness of masked video modeling is closely tied to the targets it reconstructs or aligns with.”: This statement is hard to understand without context about the training data. What are the targets? How is “higher semantic richness” introduced?  
* Results also suggest conclusions that are not discussed in the paper. These should be addressed and discussed in more detail:  
  * VidTAB: A missing insight is that VFMs show no gain over IFMs on spatial tasks (Safety and Quality).  
  * Image-to-video methods do better than VFMs on most tasks.  
* From the evaluation results on both benchmarks it is not clear what the gap between VFMs and state of the art on each dataset is since results on specialist models on each task are not given.  
* There are significant differences in results between the VidTAB and VidEB benchmarks that are not explained.  
  * Why does stage 2 training improve UMT-L and InternVideo2 performance on VidTAB but degrades performance on VidEB?  
  * Similarly, why does fine-tuning on K710 increase performance on VidTAB but hurts performance on VidEB?  
* The related work section does not discuss image foundation models and image-to-video adapter approaches. It would be good to at least discuss those approaches that are evaluated. This would allow readers who are less familiar with the literature in this area to better interpret the results.  
* Evaluation of task adaptation is limited to one method (attentive probe).  
* Future research directions are not clearly pointed out, though some ideas are given in the results discussion. A separate section on this could help direct future research.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please address all the weakness 
Also please indicate: Table 5 what models are fine-tuned and what are zero-shot. 
How are authors justifying their contribution (d) Vision Centric, line 111 "avoiding the introduction of biases that may arise from incorporating language models"
soundness: 3
strengths: (1)	**Evaluating / inferencing based on 20 models** strengthens the results.    


(2)	**Variety of tasks**, namely: action recognition, “AI for science”, “video content moderation”, “video quality/aesthetic assess” and “emotion analysis” further strengthens the findings.   


(3)	Work **addresses the shortcoming of conventional benchmarks**:  
(a)	Expanding diversity of evaluation (contrary to traditional benchmarks which mainly focus on action recognition) (8 classification tasks)   
(b)	The Diversity of the evaluation set and evaluation protocols differentiates between models, despite their saturation in performance on traditional datasets. The idea of a "TA-score" is really good and well-motivated.   
(c)	Conventional evaluation often necessitates end-to-end training (not practical for Large VFMs), thus using few-shot learning as a low-cost alternative.  
(d)	Vision-centric, focusing on vision models, discounting performance differences that may be caused because of language models.  


(4)	**Paper is well-written** and easy to understand  


(5)	VidTAB (few-shot learning with 8 tasks) & VidEB (embedding evaluation on downstream 4 tasks) both tasks provide a good comparison for models.    


(6)	Some insights namely: “augmenting video training data can sometimes negatively affect certain tasks”, and the effect of “pre-training paradigms” are interesting.
summary: Authors aim to address the evaluation of video foundation models via a comprehensive benchmark. The author introduces VideoEval to address the limitations of traditional benchmarks (and evaluation protocols), namely poor diversity, high evaluation cost, and saturated performance metrics for Video Foundation Models. Work is partitioned into (a) VidTAB (few-shot learning) and (b) VidEB (feature embedding’s direct applicability to downstream tasks). Work evaluates 20 Foundation models, revealing:   
(a) weak generalization across diverse tasks   
(b) dataset size doesn’t increase the performance   
(c) effectiveness of pretraining is understudied   
(d) combining different training pretraining paradigms helps models generalize well.
weaknesses: 1. **Clarity Needed**: How are authors defining “Beyond Action recognition” “Task Diversity” and “Domain Diversity”, e.g. VideoGLUE Yuan et al. (2023) is restricted to action recognition while it is task and domain-diverse. (Table 1).    

2. **Comparison with previous work is misleading**:   
a.	**VideoGLUE** (Yuan et al. (2023)) is restricted to action recognition only, although it consists of other tasks like temporal localization, and spatiotemporal localization, while this work counts “Medical Surgery” (medication action recognition) as separate task from normal action recognition.   
b.	**BEAR** (Deng et al. (2023)) is counted as only action recognition while it does  “anomaly (violence) classification” which this work counts as separate content moderation. Similarly, their “Instructional Video” classification is counted as a separate task as AI for Science via “Medical Surgery”. Their “Gesture” recognition is similar to “Emotion Analysis” and “Animal Behavior” which are counted as separate tasks in this work.   
c.	**VidTAB is classification only** (albeit different datasets). Temporal action localization is a much harder and a diverse task compared to classification. [Line 87-88] “Focus on action recognition, overlook other video understanding scenarios” This motivation is weak. Maybe reword this? 

3. **Figures are difficult to understand**: Figure 1 can be easily summarized and it is occupying too much space. The same applies for Figure 2. Comic San font is hard to read and difficult to differentiate what part of the figure is important and what part belongs to the setup environment. Generally, these fonts denote dataset specific / setup, while methodology / proposed technique is in default font (but its opposite here?). Table 2 (and Table 3) are unreadable. So small font size. Put it in supplementary and expand it? 

4. **Choice of words ‘Evaluation’ vs ‘Comparison’**: [Line 020-021] “*evaluating* the task adaptability of VFMs under few-shot conditions”. Benchmarks traditionally evaluate the models on a variety of tasks. Comparison among models is the byproduct of these benchmarks.  
(a)[line 232-233] “exclude datasets with videos that have low resolution”,   
(b) [line 250-251] “the number of categories is too high, models often perform no better than random guessing”.  
These preprocesses are mainly done for model comparison and not in the spirit of true evaluation. 

5. **Authors seemed to have missed the technique of “Visual Prompt Tuning”** (Jia, Menglin, et al. "Visual prompt tuning." European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.) for “Identifying efficient adaptation method for evaluation”? It just adds 50 spatial tokens to adapt to models for various fine-tuning tasks. 


6. **I’m doubtful if [Line 107] Low-cost is truly a contribution**. This study deals with foundation models which are especially good in zero-shot evaluation. It's unfair to count it as a contribution when dealing with such models, and comparing traditional benchmarks with traditional models. Additionally, What’s “’K710ft” Kinetics-710 (0.66M) is this few shot? 0.66M is not low cost as indicated in Figure 1.  


7. **Space allocation is unconventional**. For a benchmarking paper, the setup for benchmarking takes the majority of space, while the benchmarking analysis starts at the 8th page (out of 10 pages) with minimal analysis. It's not well-reasoned why image models are included when benchmarking video models. 


8. **[LINE 424-425] “The adaptation performance of models generally increases with the growth of data and model size” This is not necessarily true and requires a deeper analysis**.   
(a).	Within similar models (i) “VideoMAEv1-H” has 651 M parameters with superior performance to “VideoMAEv2-g” (1037M). (ii) “ViCLIP-L-200M” (200 M pretraining) is inferior to ViCLIP-L-10M (10 M pretraining) on “Harmful Content”, and “Quality Assess”.   
(b).	Outside the model the number of contractions is far greater. E.g. VideoMAEv2-g < UMT-Lstage1.