VideoGLUE: Video General Understanding Evaluation of Foundation Models
Authors: Liangzhe Yuan
Nitesh Bharadwaj Gundavarapu
Long Zhao
Hao Zhou
Yin Cui
Lu Jiang
Xuan Yang
Menglin Jia
Tobias Weyand
Luke Friedman
Mikhail Sirotenko
Huisheng Wang
Florian Schroff
Hartwig Adam
Ming-Hsuan Yang
Ting Liu
Boqing Gong
Venue: Accepted by TMLR ()

Abstract: We evaluate the video understanding capabilities of existing foundation models (FMs) using a carefully designed experiment protocol consisting of three hallmark tasks (action recognition,temporal localization, and spatiotemporal localization), eight datasets well received by the community, and four adaptation methods tailoring an FM for downstream tasks. Furthermore,we jointly profile FMs’ efficacy and efficiency when adapting to general video understanding tasks using cost measurements during both training and inference. Our main findings areas follows. First, task-specialized models significantly outperform the seven FMs studied in this work, in sharp contrast to what FMs have achieved in natural language and image understanding.  Second, video-native FMs, whose pretraining data mainly contains the video modality, are generally better than image-native FMs in classifying motion-rich videos,localizing actions in time, and understanding a video of more than one action. Third, the video-native FMs can perform well on video tasks under light adaptations to downstream tasks (e.g., freezing the FM backbones), while image-native FMs win in full end-to-end finetuning. The first two observations reveal the need and tremendous opportunities to conduct research on video-focused FMs, and the last confirms that both tasks and adaptation methods matter when it comes to the evaluation of FMs. Our code is released under: https://github.com/tensorflow/models/tree/master/official/projects/videoglue

Published in Transactions on Machine Learning Research (10/2024)
VideoGLUE: Video General Understanding Evaluation of Foundation Models
Liangzhe Yuan*i
Nitesh Bharadwaj Gundavarapu"
Long Zhao"
Hao Zhou"
Yin Cui Lu Jiang
Xuan Yang
Menglin Jia
Tobias Weyand
Luke Friedman
Mikhail Sirotenko
Huisheng Wang
Florian Schroff
Hartwig Adam
Ming-Hsuan Yang
Ting Liu
Boqing Gong
Google DeepMind
Reviewed on OpenReview:
Abstract
We evaluate the video understanding capabilities of existing foundation models (FMs) using carefully designed experiment protocol consisting of three hallmark tasks (action recognition, temporal localization, and spatiotemporal localization), eight datasets well received by the communily, and four adaptation methods tailoring an FM for downstream tasks. Purthermore, we jointly profile FMss efficacy and efficiency when adapting to general video understanding tasks using cost mcasurements during both training and inference. Our main findings as follows. First, task-specialized models significantly outperform the seven FMs studied in this work, in sharp contrast to what FMss have achieved in natural language and image understanding. Second, video-native FMs, whose pretraining data mainly contains the vidco modality, are generally better than imago-native FMs in classifying motion-rich videos, localizing actions in time, and underslanding a video of more than one action. Third, the nativee FMs can perform well on video tasks under light adaptations to downstream tasks (c.g., freezing the FM backbones), while image-native FMs win in full end-to-end finetuning. The first two observations reveal the need and tremendous opportunities to conduct research on video-focused FMs, and the last confirms that both tasks and adaptation methods matter when it comes to the evaluation of FMMs. Our code is released under https:
1 Introduction
Foundation model (FM) is a term coined by [oommasani et al. (0021), referring to "any model that is trained broad data, that can be adapted (c.g., finctuned) to wide range of downstream tasks." Some representative FMs include but are not limited to BERT (Devlin et al. 2018), GPT-3 (Brown al. 2020 CLIP (Radord et al. 2021), and ALIGN (Jia et al. 2021). This work primarily investigates the video understanding
capabilies of seven visual and multinodal PMs: CLIP (Radford et al.] 2021). FLAVA (Singh et al. 2022
CoCa YYu et al. 2022) DINOv2 (Oquab et al. 2023). VATT (Akbari et al. 2021), VideoMAE (Tong et al.
2022), and Intern ideo (Wang et al.] 2022b)
We select these models because they are amendable for the
video understanding and make their checkpoints accessible to us.
"Equal technical contributions.
*Corresponding lzyuandgoogle. and bgongogoogle.
Work done Google. YC is NVIDIA; I.J is now ByteDance; MJ an intern Google and is Meta.

---

Published in Transactions on Machine Learning Research (10/2024)
MANK
MANK
(STAL)
(VCA)
1STALL
(VC-)
(VCA)
(CC4)
(TAL)
TVCAJ
(SALI
(TAL)
(VC-) (TAL
VCA)
53+2
Chanadia
2544
Charades
53+2
Char ades
Cha reredd
(VC
(VCM)
MC MI
(VC-M)
(a) CLIP
(b) FLAVA
(c) CoCa
(d) DINOv2
HANK
ESTAL1
ISTAL
(V-A)
OSTALL
(VC-)
ISTRAL
(VCA
Atve
NCCII
(TALL
(YCA)
(CCA
50W2
Characes
5342
Charades
50xd
Charuces
(VC
(VCM)
(VCM)
VATT
(I) VideoMAE
(g) InternVideo
Figure 1: Performance of FMs with end-to-end finetuning (red) and frozen backbone (blue), in comparison with trate- task-specialized models (black) on VideoGLUE benchmarks. VC-A, VC-M, and VC-ML stand for appearance-focused, motion-focused, and multi-label Video Classification tasks, respectively; TAL stands for Temporal Action Localization; STAL stands for Spatiotemporal Action Localization. The highest and lowest performance numbers on each dataset are mapped to 0.9 and 0.1, and the other numbers are linearly scaled accordingly on the radar chart. We also use gray shades to represent tasks that are more focused on appearance understanding more than motion. We observe that: (1) FMs generally fall behind task-specialized models; (2) FMs that are trained with video data are generally better than image-native FMs on motion-focused focused tasks under the frozen backbone setting, and image-native FMs can generally catch up when finetuned end-to-end on the target dataset.
It is nontrivial to evaluate FMs. In contrast to "specialist" models developed for a particular task, FMs are considered as "generalists" that learn sharcable meta-knowlooge across tasks so that one can quickly adapt them to achieve superior performance on various downstream lasks. Hence, both the tasks and adaplation methods matee when it comes the evaluation of FMs. However, the community has not reached a consensus on these two aspects. FM developers select their own different sets of downstream tasks interestingly, often covering no video only appearance-rich video classification tasks (Buch et al. 2022 Lei et al. 2023) Moreover, they rely on distinct adaptation methods, making apples-lo-apple comparisons challenging and causing mismatches with the FMs' actual use cases.
To this end, we propose to evaluate FMs' video understanding capabilities using a carefully designed experiment protocol, named VideoGLUUF, consisting of three hallmark tasks (action recognition, temporal localization, and spatiotemporal localization), eight datasets well received by the rescarch community, and four model adaptation methods tailoring foundation model for downstream tasks. The tasks examine an FM from various aspects needed for understanding video. The "all-around" adaptations represent the main cases of FMs in the literature and, more importantly, allow us to thoroughly probe an FM's potential in video understanding.
Why do we specifically focus on videos? The main motivation to promote video understanding in the ovaluation of FMs. More coneretely, we test the following conjectures through this work. First, FMs' high performance on existing evaluation suites does not necessarily indicate their potential in video since these suites either lack video-specifie tasks selectively choose video tasks whose appearance feature more
2

---

Published in Transactions on Machine Learning Research (10/2024)
important than motion - InternVideo (Wang et al. Second, many existing FMs are unlikely to be able to handle motion in video well, given that they learn primarily from static images (Radford et al. 2021
2022b is an exception as discussed in the next paragraph.
Singh et al. 2022 Yu et al. 2022) or short video clips containing limited motion [Feichtenhofer et al. 2022 Wang ct al. 2022b). Third, popular adaptation mothods
(e.g,, finetuning all weights) cannol supplement FMs with all the cues needed to recognize motion-rich actions
and localize entities temporally and/or spatiotemporally, as elaborated in Sections 4.1 and 4.2
While work is not the first to emphasize the evaluation of FMs, it is unique on multiple fronts. Unlike ELEVATER (Li et al. 2022a)'s target of evaluating aanguage-angmented FMs, we consider all FMs adaptable to video understanding which does not necessarily involve language. Unlike Perception Test (Patrauccan) et al. 2024h's i coverage of a broad spectrum of perception tasks, we focus on video, allowing us to cover various aspects of this vertical domain. Interestingly, many of our datasets also appear in InternVideo (Wang ct al. 2022b), a video-oriented FM. However, we promote model adaptation methods as an inherent part of the evaluation protocol a consistent set of diverse adaptation methods is nocessary to provide FMs ample opportunities to expose their video understanding capabilities. Moreover, unlike InternVideo's 's focus on their single FM, evaluate FMs developed by different research groups in an uniform experiment protocol the first of its kind for visual and multimodal FMs, to the best of our knowledge.
Our main findings are as follows. First, lask-specialized models still significantly outperform the seven FMs studied in this work (see Figure I in sharp contrast to what FMs have achieved in natural language (OpenAl
2022; Roberts et al. 2022) and image understanding (Radford et al. 2021 Yu et al. 2022 Chen et al. 2022 Hence, there is a need and tremendous opportunities to research Aideo-focused FMs. Second, video-native FMs, whose pretraining data mainly contains the video modality, are generally better than image-native FMs in classifying motion-rich videos, localizing actions in time, and understanding video of more than one action. Third, the video-native FMs can perform well on video tasks under light adaptations to downstream tasks (e.g., freezing the FM backbones), while image-native FMs win in full end-to-end finetuning. This observation confirms that both tasks and adaptation methods matter when it comes to the evaluation of FMs.
2 Related work
Foundation models. One common type of FMs Large Language Models (LLMs) trained to acquire
generic, transferable, and diverse representations that can enable sample-efficient learning and knowledge transfer across broad range of downstream tasks. FMs are often trained with simple self-supervised learning objectives such as predicting the next token in a sentence (e.g., GPT-3 (Brown et al. 2020). PaLM (Chowdhery ct al., 2022)), ), or denoising the masked tokens (c.g., BERT (Devlin et al.) 2018). UniLM (Dogg ct al. 2019), and BETT (Bao el al., 2. An intriguing characteristic of FMs is their ability to gradually acquire new capabilities as the model grows and the training data size increases, despite being trained on simple learning objectives (Wei et al.) 2022). For example, PaLM (Chowdhery et al.) 2022 Anil et al.) 2023 massive LM with 540 billion parameters, has started to show new capabilities in tasks such as explaining jokes, solving math, and performing common-sense reasoning when scaled to over 100B parameters.
In addition to self-supervised transformers, FMs in computer vision also encompass transformers specifically trained to align image-text text paired data. These FMs use learning objectives include contrastive learning (e.g., CLIP (Radford et al., 2021)), ), denoising masked tokens (e.g., BEiT-3 (Wang et al. 2022a)), predicting the next token in single modality (e.g., DALL-E (Ramesh et al., 2021 ) or in the interleaved image-text sequence (e.g., Flamingo (Alayrac et al., 2022). Kosmos-1 (Huang et al. 2023 ) Recent FMs are also trained a mixture of these objectives (e.g., MAE (He et al. 2022), FLAVA (Singh et al. 2022). and CoCa (Yu et al. 2022 ). Our criteria of choosing foundation models to study are primarily based on the definition of FMss and their amendability on video underslanding and accessibility of checkpoints. We leave some models (e.g,, detection, segmentation models) out of the scope of this work, because of their current lack of generalization on vidco understanding tasks. Finally we choose seven representative FMs, i.c., CLIP (Radford et al., 2021). FLAVA (Singh et al.) 2022), CoCa (Yu al. 2022), DINOv2 :Oquab et al. 2023 VATT (Akbari al.
2021 VideoMAF (Tong et al. 2022), and InternVideo (Wang et. al. 2022b).
3

---

Published in Transactions on Machine Learning Research (10/2024)
Table Foundation models studied in this work (MxM stands for Masked Image/ Lanauggg /aiggueege///eoooooo Video Modeling).
Foundation Model
Pretraining Modality
Pretraining Data
Pretraining Objective
CoCa
Yu et al.
2022
Image + Text
JFT3B ALIGN
Coutrustive Captioning
CLIP
[Radford et al.
2021
Image + Text
WeblmageText
Contrastive
FLAVA
Singh cf al.
2122
Image + Text
PMD
Contrastive + MIM + MLM
DINOv2
Oquab et. al
2023
Image
LVD-142M
MIM + DINO
VideoMAE (Tong
al. 2022
Video
K400
MVM
InternVideo [Wang et
2022b
Video
UnldbeledHlybrid
MVM + Contrastive
VATT Akbari et al.
2021
Video + Audio + Text
HT100M
Contrastive
Evaluation of foundation models. As the mission of FMs is to enable sample-cfficiett knowledge transfer, the design of downstream lasks critical to evaluate the capabilities and linitations of these models. The evaluation of FMs is pioneered by the NLP researchers. For example, GLUE (Wang et al. 2018a) and SuperGLUE (Wang et al. 2019) introduced a suite of tools for evaluating language understanding tasks. The authors utilized established public benchmarks and provided tools for evaluating, probing, and benchmarking pretrained FMs, allowing for a comparison to human baselines. ELEVATER (Li et al. 2022al introduced this concept to vision FMs along with a toolkit for evaluating lasoun- tasks, including knowledge augnentation, hyperparameter tuning, and three adaptation techniques. In parallel, there have been attempts to establish diagnostic benchmark for perceptual understanding of the world. For instance, Perception Test (Patraucean et al., 2024 crowd-sourced 11K videos in which about 100 users performed scripted activities. This benchmark comprises videos filmed by only about 100 participants, which may not provide the same level of domain coverage and diversity as the other FM evaluation works mentioned earlier.
Evaluation of video foundation models. While some vision-language FMs have incorporated video tasks, their evaluation typically follows that of static images and neglects the unique aspects of video spatial-temporal modeling and reasoning. To our knowledge, no previous work has been solely dedicated to evaluating video FMM. The closest work to ours are InternVideo (Wang et al., 2022b) and VideoMAE (Tong et al. 2022), which introduce new FMs and show their superiority over several video datasets. This work has two key differences to the prior ones. First, evaluation is video-centric using the tasks that require motion understanding or long-term temporal reasoning. Sccond, instead of promoting new vidco FMs, our work proposes no new models and is solely dedicated to evaluating current and future video FMs in an impartial reproducible experimental setup. Concretely, our goal is to provide tools for probing and benchmarking FMs on motion tasks in various settings.
3 Tasks and adaptation methods both matter when evaluating foundation models
This section describes our video general understanding evaluation (VideoGLUE) benchmark. We first introduce the visual and multimodal FMs evaluated in this work. Then we discuss the video-focused downstream tasks and methods adapt an FM the tasks. The former concretizes the video understanding capabilities want to evaluate from an FM, while the latter provides various paths for an FM to showcase the corresponding capabilities.
3.1 Foundation models for video understanding
We are interested in examining which FMs are good at solving video tasks, what makes them better than others in the video domain, and how to best adapt them to video understanding. Table [ shows the seven FMs we gained access to via public repositorics personal communications. Thanks to the powerfulness and scalability of the Iransformer architecture (Vaswani et al. 2017), most developed FMs converge to adopt the vision transformer architecture. Thus for all evaluated FMs, we intentionally choose the ViT-B (Dosovitskiy 2020) variant to bring fair comparison into our benchmark. We also notice, previous litcrature, models may be evaluated with different number of frames and resolutions, resulting in unfair comparison Yan 2022 Feichtenhofer et al. 2021). In VideoGLUUF, we control the number of tokens observed by the et al.
ct al.
4

---

Published in Transactions on Machine Learning Research (10/2024)
Table 2: Summary of statistics, video properties, and data sources of each dataset. Tasks involved are video classification (VC), spatiotemporal action localization (STAL), and temporal action localization (TAL).
Task
Dataset
# of videos (train/viil /validation)
Avg. length
Source
Notes
Kinetics-400
235, 693 19, 165
10 secs
Web
Holistic, appearance
Moments in Time
791. 246
33, 898
3
Web
Holistic, appearance
VC
Something-Someehing
168, 913
24,777
6 soes
Crowdsourec
Holistic, motion
Diving48
15, 027 1,970
Web
Holistic, motion
Charades
7,811 1,814
30 sers
Crowdsource
Multi-label, long-elip
TAL
ActivityNet v1.3
10, 002 / 4,926
5 ~ 10 mins
Web
Temporal
AVA v2.2
210, 634
57,371
15 mins
Movic
Spatiotemporal, instance
STAL
AVA-Kinetics
354, 201 91,919
10
Web
Spatiotemporal, instance
ViT to be consistent for different FMs on the same task. For the detailed setup on each dataset, please refer to supplementary materials D
3.2 Video understanding tasks
Like objects' role image understanding, actions are the core of video understanding, leading us to select tasks and datasets that recognize and localise actions in time and space. Table 2 provides quick summary. Next, we explain the rationale behind the particular choices of datasets and postpone the datasets' details to the supplementary materials
3.2.1 Recognizing actions
General actions. We first include the action recognition datasets of Kinetics-400 (K400) Kay et al. 2017 Moments in Time (MiT) (Monfort et al. 2019), and Charades (Sigurdsson et al. 2016 considering their popularity and they are complementary to each other. Data domain coverage is an important factor when designing benchmarks for FMs, as nowadays FMs are typically trained on massive data sources. K400 videos are from YouTuue, MiT draws videos from different Web venues, while Charades contains scripted indoor videos. Internet often returns entertaining and atypical videos, while Charades is about typical everyday videos (Sigurdsson et al., 2016). Regarding action labels, the datasets differ in gramularities and real-life scenarios: verb defines action in MIT, K400 groups actions by verb-subject pairs, and Charades actions are about indoor activities. Regarding the average length, K400 and MiT videos are between 3 and 10 seconds, each with one action label, while Charades videos are about 30 seconds, each with multiple actions.
Finc-grained motion-focused actions. We also include fomething-somethiin v2 (SSv2) (Goyal et al. 2017) and Diving48 (D48) (Li et al. 2018) as another two action recognition datasets, whose actions are fine-grained and motion-focused. SSv2 contains 174 human hand gestures as action labels, such as putting something into something, turning something upside down, and covering something with something. The videos are mostly focusing hand-object interactions. D48 videos all about competitive diving recordings collected from Web sources. Notably, in these datasets the foreground objects' motion is a more significani. discriminative cue than their appearance.
3.2.2 Localizing actions
The vidcos in action recognition are trimmed, but actions could occur anywhere in video in the wild. Hence, temporal and spatiotemporal action localization is also crucial to video understanding, Accordingly, we
choose three datasets for the experiments: the action localization track ActivityNeet v1.3 (ANet) Fabian Caba Heilbron & Nicbles 2015), Atomic Visual Actions (AVA) (Gu et al. 2018), and AVA-Kineties (AVA- K) (Li et al.) (022)). The last two require model localize and recognize actions in both time and space, and their underlying videos movies and general YouTube videos, respectively.
5

---

Published in Transactions on Machine Learning Research (10/2024)
Target
Target
Target
Query
Target
Tuned
Token
Layer
Layer
Layer
Leyer
Patch
X-Artn Layer
.1.
Encoder Layer
*
Layer
Layer
Adapter Layer
X-Artn Layer
Encoder Leyer
* Encoder Layer
*
Encoder Leyer
Adepter Encoder Layer
Layer
Encoder Layer
Layer
Adapter Layer
Encoder Layer
g
Encoder Layer
* Encoder Layer
Adopter Encoder Layer
(c)
(d)
Figure We study four adaptation methods to apply foundation model (FM) to video understanding downstream tasks: (a) end-to-end finetuning, (b) frozen backbone, (c) frozen backbone with multi-layer
attention pooler (MLAP), and (d) low-rank adapter.
3.3
Adaptation methods
In this section, we detail the task-specific neural architecture design and adaptation methods when applying FMs downstream tasks.
3.3.1
Modifying foundation model architectures for downstream tasks
Given an FM(-), we can apply FM(-) to vidco clip C' to extract a set of feature maps {F)* =M(C), Fe Rexhxuxe, where k is the number of endpoint layers from an FM, and n, h, w, are respectively feature map's length, height, width, and number of channels.
For video classification tasks, we cast feature map F as n X h x w tokens and aggregate them into a global representation using a learnable query loken and lightweight cross-allention layers ( (Dosovitskiy et al. 2020 For spatiotemporal action localization, following the standard practice (Feichtenhofer et al. 2019 Tong et al. 2022), we first detect humans on key-frames using a human detector (Ren et al.) 2015). producing a set of human bounding boxes B. We then apply the Rol pooling operation (Jaderberg al. 2015) that takes both the feature map F and box coordinates B as inputs and outputs leature vector per box the query token, RoIPooL(F. F, B), followed by the same cross-attention layers in video classification. For both groups of tasks, we stack linear classifier top of the task token's 's last-layer encoding for final classification:
= LINEARCLASSIFIER (CROSATTTTTIOON
(1)
For temporal action localization, we first perform feature extraction in sliding window manner, resulting in sequence of globally average pooled features /AvGPooL. (FF))oFFFF.....)) (Fi),... AvGPoor. (F)) for each video. Following
popular choice of prior works (Alwassel et al. 2021 Ju et al. 2022 Liu et al. 2022). we employ G-TAD (Xu
et al. 2020 as our task head for predicting the action category and its start and end timestamps.
3.3.2 Adapting modified foundation model to downstream tasks
Adapting the modified FMs to a downstream task to tune their weights. Then, we immediately have two basic adaptation strategies: (1) full finetuning to update all weights in the original FM plus the task head and (2) froczing FM weights and only updating newly added weights. The choice of the adaptation methods depends on specific application scenarios such computation and memory constraints. We argue that an ideal FM should perform well across various adaptation methods to support. the breadth of use cases.

---

Published in Transactions on Machine Learning Research (10/2024)
End-to-end finetuning. End-to-end finetuning is the most common FM evaluation method for videos
(Akbari et al. 2021: Feichtenhofer et al. 2022: Tong et al. 2022 Wang et al. 2022b), but it requires the deployment of a separate and possibly expensive FM for each downstream task. When finetuning all weights in the modified FMs, we limit cross-attention to a single transformer layer with 12 heads and hidden size 768.
We vary learning rates and weight decays for each experiment to ensure every FM is configured to its best. setup. Figure aaa) illustrates this end-to-end finetuning.
Freezing foundation model weights. Linear probing and cross-attention based pooling over frozen FM features are routinely used to test the strength of the FM representation (Tong et al, 2022 Yu et al 2022 Singh et al., 2022 He et al. 2022 Lin et al. 2022). In practice, adapting task-specific heads with frozen FM allows us to deploy the same FM for multiple tasks. If we use light-weight heads over the FM features, then a single FM inference can serve multiple tasks efficiently in terms of both compute and memory. To this end, we examine two variations with a frozen FM, one with a single cross-attention layer and the other with multiple layers. The first results in exactly the same model architectures as in end-to-end finctuming (Figure b))) and the second allows us to leverage an FM's hierarchical features beyond its last endpoint layer (Figure (c)). First, the frozen features are extracted from the last k layers, FN-4+1, FN-K+2, .... Fx. Then, attention pooling is applied between a Icarnable token 7 and the fcatures FN-K+ using multi-head cross-attention (MHCA). The output of this layer serves as the query token for the next round of attention pooling with the features F This process is repeated for le rounds:
TN-A+1 MLP(MMCA(() Fx-1F)))))
TN MLP((HCA(( F FN-
(2)
TN = MLP(MHCA(XX-I,F))
where k = 4 in our experiments, and the final classifier is p = LINEARCLASSFFIR(TT)))
Freezing foundation model weights with low-rank adaptation. Finally, we explore a frozen FM beyond the last h layers using low-rank adapter (Hu et al., 2021), which is bottleneck architecture that projects a feature tensor into a low-dimensional space and then up-samples to the original space. The bottleneck space's ss dimension is 64 in our experiments. Through inserting few adapter layers with trainable weights {w} into the pretrained FM while keeping all FM's weights frozen, the feature adapter is more paramcter-officien than end-to-end finetuning the whole network while achieving better performance than simply adding a task head to the frazen FM. Essentially, the adapter leads to a new FM with some trainable weights {w]: F = FM {w))), such that the output feature maps remain the same in shape as the original FM's output (Figure dd). Hence, different pooling schemes and task heads aforementioned could be applied to the extracted feature map F. For simplicity, we still choose the single-layer cross-attention as the default. task head due to its computation efficiency and performance.
The low-rank adaptation allows single FM for multiple tasks, in contrast to the per-task models in end-to- end finetuning. However, it incurs a per-task forward pass at inference time, being less efficient than the task-specific heads over frozen features.
4 Experiments
4.1
End-to-end finetuning
Table shows the end-to-end finetuning results of six FMs on eight datasets. We split the FMs into two groups based on their input modalities at the time of pretraining: CLIP, FLAVA, CoCa, and DINOv2 are image-native FMs, while VATT, VideoMAE, and InternVideo are video-native. The datasets span video classification (VC) and spatiotemporal action localization (STAL). Note that it is infeasible to end-to-end fine-tune or LoRA fine-tune the vision encoder on TAL lask, because the videos in ANet are typically long (up to 30 minutes). We follow the common practice of pre-computing visual features in a sliding window manner offline, and training a temporal detection network on top of the visual features (Wang et al. 2021 Zhang et al. 2022). We will report TAL results in the next section. We draw the following observations from
Table g
7

---

Published in Transactions on Machine Learning Research (10/2024)
Table 3: Evaluating FMs when adapted to video understanding tasks using end-to-end finetuning. We report the Top-1 accuracy on K400, MiT, SSv2 and D48, MAP on Charades and ANet, and mAP@IOU0.5 on AVA and AVA-K.
VC-A)
VC-M
VC-ML Charades
TAL ANet
STAL
Models
Avg.
K400
MiT
SSv2
D48
AVA
AVA-K
CLIP
81.0
39.0
46.6
75.7
54.3
27.1
28.9
52.8
FLAVA
79.1
38.3
61.1
72.0
48.6
22.0
25.6
49.4
CoCa
82.6
43.6
66.8
79.6
55.0
27.7
31.0
55.2
DINOv2
80.3
38.7
60.9
70.2
52.4
24.6
27.0
50.5
VATT
77.1
34.8
65.1
77.6
55.7
27.0
28.4
52.7
VideoMAE
78.7
36.1
65.5
75.5
51.4
23.5
26.2
51.0
InternVideo
80.1
35.9
67.0
75.8
52.2
27.2
29.8
52.5
Task-
88.6
42.7
68.7
88.9
63.2
37.5
42.3
38.9
specialized
(TubeVIT)
(UniformerV2)
(MVIT)
(AIM)
(MoVINet)
(PRN)
(RAFT)
(RAFT)
FMs underperform task-specialized models on video tasks in generol. Table 3 last row collects the statc-of-tho- art results on the eight datasets, each obtained by a task-specialized model with comparable architecture or size to ours the prior work. Specifically, those task-specialized models are RAFT (Rajasegaran al. 2023) PRN (Wang al., 2021), TubeViT Piergiooaani et al. 2023), Uniformer V2 (Li al 2022b), AIM (Yang et al. 2023 MVIT (Fan et al., 2021 and MoViNet (Kondratyuk et al.] 2021 respectively. All seven FMs underperform the task-specialized models on all video tasks except on Moments in Time at the comparable model scale, indicating the lack of strong vidco-focused FMs. This observation is in sharp contrast to what FMs have achieved on natural language (OpenAl 2022
Anil et al.) 2023) and image understanding (Chen
et al. 2022).
CoCa performs the best among image-native FMs on the video tasks. It actually gives risc the highest accuracy all datasets, with slightly inferior performance generalization capability of the CoCa model, regardless it is an image-based model with image-only pre- training data. However, in the latter session, we will reveal that under light-weight and parameter efficient adaptation scenarios, the same model may perform inferior on many video understanding tasks, especially on SSv2 (Tables 5 and Charades (Tables
SSv2 and Charades. This shows strong
and () and ANet (Tables 4 and 5], which require complex motion or multiple actions understanding per video. These are in contrast that CoCa achieves the best general performance in end-to-end fino-tuning (Table highlighting the importance of considering adaptation methods FMs benchmarking.
4.2 Freezing foundation models
End-to-end finetuning is infeasible for some application scenarios due to FMs' rapidly growth in size and the consequent demands in computational resources. In the following, we evaluate frozen FMs with various adaptation methods. Tables 4 5 and f are the results of adaptation with a single cross-attention layer, multiple cross-attention layers, and a low-rank adapter, respectively.
Generally speaking, DINOu2 performs the best in the frozen feature pooler evaluation (Tables 4 CLIP performs the best with meali--head allention pooler aranng image-native frozen FMs (Tables 5) but CoCa catches up thanks to the low-rank adapter (Table Q). It is worth noting that this ranking of intive frozen FMs differs from the ranking of image-native FMs in end-to-end finetuning. It seems that DINOv2 and CLIP's frozen features are more amendable to the video tasks than CoCa, but CoCa as a whole adapts better to video under both finetuning and the adapter. Hence, it is crucial to consider adaptation met hods as an organic part of the evaluation of FMs to supply them various paths to demonstrate their capabilities.
Video-native FMs are better than image-native FMs in understanding motion-rich SSe2 and D48, Charades that contain mullippe actions per video, and ANet for temporal action localization. This observation is
8

---

Published in Transactions on Machine Learning Research (10/2024)
Table 4: Evaluating FMs when adapted to video understanding using frozen features. Only weights in the task heads are updated using the downstream tasks' training sets.
VC-A
VC-M
VC-ML Charades
TAL
STAL
Models
Avg.
K400
MIT
SSv2
D48
ANet
AVA
AVA-K
CLIP
75.2
32.6
41.0
44.1
11.2
32.7
21.1
25.9
32.8
FLAVA
71.3
29.7
40.6
45.9
12.6
32.2
18.8
21.5
31.7
CoCa
32.0
41.5
34.1
8.8
33.0
24.7
31.2
DINOv2
72.3
29.0
40.0
40.4
25.8
32,6
24.6
26.9
35.0
VATT
32.1
57.8
49.7
33.3
35.3
20.3
22.2
39.1
VideoMAE
65.1
23.0
53.9
59.5
11.3
33.0
16.0
19.9
32.6
IntervVideo
69.3
26.3
58.2
55.6
13.0
33.3
13.4
15.7
33.1
Table 5: Evaluating FMs when adapted to video understanding using multi-layer attention pooler (MLAP), which takes multiple frozen features from an FM as inputs and map them hierarchically for the final task prodiction. Only the multi-layer attention pooling layers are updated using the downstream tasks' training sets.
VC-A
VC-M
VC-ML Charades
TAL ANet
STAL
Models
Avg.
K400
MIT
SSv2
D48
AVA
AVA-K
CLIP
77.1
39.0
50.1
55.8
41.5
33.9
27.7
29.6
43.3
FLAVA
71.5
34.5
43.1
58.5
38.2
32.4
21.3
23.2
39.3
CoCa
74.2
37.2
45.9
48.4
19.6
33.3
27.0
36.3
DINOv2
75.4
36.0
46.3
51.9
47.8
33.6
25.4
27.0
42.5
VATT
35.6
58.7
60.1
58.2
35.0
22.9
46.3
VideoMAE
71.7
32.2
37.4
69.6
35.9
33.4
19.6
22.1
40.9
InternVideo
73.7
34.7
60.3
71.9
40.5
33.6
15.9
17.7
42.2
Table 6: The low-rank adapter results of FMs for video understanding. We only update the weights of the adapter and task head while keeping the original FMs' weights frozen.
VC-A
VC-M
VC-ML Charades
TAL
STAL
Models
Avg.
K400
MIT
SSv2
D48
ANet
AVA
AVA-K
CLIP
80.2
39.7
56.0
77.2
44.2
24.5
28.0
49.3
FLAVA
74.7
34.1
52.1
68.4
40.8
23.8
44.1
CoCa
80.9
41.4
56.1
67.1
45.8
26.6
28.7
49.0
DINOv2
77.7
36.1
59.0
76.6
38.0
22.5
27.9
47.0
VATT
75.0
36.5
63.5
68.9
53.5
22.3
25.8
49.9
VideoMAE
73.6
30.6
61.4
76.0
43.0
16.6
23.3
45.9
IntervVideo
31.3
63.9
73.6
46.2
25.5
47.7
about the same as the one under end-to-end finctuning. The imago-native FMs are mainly superior on appearance-rich video datasets, where high-quality spatial perceptual features are the key. We conjecture that the vast image data empowering image-native FMs is more diverse in appearance than videos used to pretrain vidco-native FMs.
Given frozen FMs, the low-rank adapler oulperforms cross-allention layers, and mullippe layers of cross- attention is better than single pross-attention layer. Many works (Caron et al.) 2021 He et al. 2022) have shown features from different layers of vision transformer have different attention maps. Hence, it is potentially beneficial have an adaptation method to leverage multiple layers of frozen FM. Table reports the results with four cross-attention layers, whose average score per model (across different. columns) is

---

Published in Transactions on Machine Learning Research (10/2024)
100
977.
Vieesuu Scoee (405)
Thainate Parsmeter
PEP-5
6o
Inteencee FL.
5
40-
[25.8]
39.3
68.6
9.7
38-
628
Frozen Adaptation Method
Adapter
37-
CLIP
FLAVA
Coca
DIMOv2 Model
VATT
VicesMA Interviido
(a)
(b)
Figure 3: (a) We measures the training (red diamond) and inforence (blue square) cost of different adaptation methods terms of number of trainable parameters and inference FLOPs, respectively. (b) We report VideoGLUU Score that combines a FM's performance weighted by its training costs with different adaptation methods for all the image-native (red circle) and video-native (blue pentagon) models.
higher than that with a single cross-attention layer (Table 4 by 18% to 40%. The low-rank adapter (Table further improves upon the cross-allention results partially because it explores all layers of frowen FM.
On average, image-native FMs outperform video-native PMs under end-to-end finetuning and the adapter, but it becomes the inverse in the other two adaptation methods. The adapter experiment paired with end-to-cnd finetuning experiment reveal the fact that existing image-based FMs could be more easily adapted to video tasks when we could adjust the feature space of FMs, possibly caused by the large-scale higher quality image(-text) pretraining datasets. On the other hand, frozen feature experiments discussed above present us the inverse picture where video- based FMs perform better. The seemingly paradox encourages more future research on bridging the gap on video-based pretraining with high-quality data, more effective modeling and better design on video benchmarks.
4.3 Profiling foundation models for video understanding
In this scction, we consolidate our studics of the FMs with different adaptation methods and vidco tasks, focusing on their overall efficacy and efficiency. Specifically, we use trainable parameters and inference FLOPs to approximately represent the training and inference costs of an FM. Since all FMs in evaluation are ViT-B and align the same number of input tokens for each task. The models have almost the same cost in one adaptation method. The left of Figure 3 shows the cost values for each adaptation method. Note that an FM with LoRA adaptor tuning could have high inference cost despite lower training/ /at///aapationnn costs than end-to-end fine-tuning. While the figure provides a holistic view of an FM from multiple dimensions, one might be interested a ranking among the FMs in terms of their video understanding capabilities. To this end, we summarize the multi-dimenssonal comparisons across different datasets, adaptation methods, and costs using simplified scalar measure, termed VideoGLUU Score (VGS), to probe an FM's general video understanding capability.
We use the cost values to normalize an adapted FM's average performance over all tasks. Formally, denoting by S, an FM's average performance score over our vidco tasks undor the i-th adaptation method and by C the corresponding cost value under the k-th developmental scenario, we calculate the FM's VGS* by
N
A and A logi0 C
VGS
Σ S. where w
(3)
ビØ=1 A
where N = 4 is the number of adaptation methods, and E [0, 1] weighs score S; according to the cost C The final VGS is the arithmetic average on {VGS*), J, where k = 1,2 corresponding to training and inference respectively.
On the right panel of Figure 3. we plot cach FM's VidcoGLUE Score. We notice the average VGS for video-native and image-native FMs on our video understanding tasks are 40.67 vs. 38.84 respectively. Zooming in to the individual FMs, we find that VATT, video-native FM, is at the first place with VGS 43.97, followed
10

---

Published in Transactions on Machine Learning Research (10/2024)
by the image-native DINOv2 with VGS 40.17. This suggests that in-domain pretraining yields overall the best adapiation capability to video tasks, and image-native FMs could also achieve competitive results on many but not all video understanding tasks.
5 Limitations
VidcoGLUE serves a comprchensive benchmark for studying and probing various video understanding capabilities of foundation models. The current (ask portfolio includes various unimodal action understanding tasks. We believe the scope of this work could be further extended as there are many other important video tasks not covered here, e.g. object or point-level tracking, long-term memory and forccasting. Moroover, our benchmark could be strengthened by adding multimodal tasks like video captioning and question answering, given the rise of general Vision Language Models (VLM). We chose three representative FM adaptation methods and used them to provide uniform experiment protocols for different FMs as possible. However, some of our observations could be flipped with the evolution of FMs development and adaptation methods, which are an active research area. We proposed a scalar score, VideoGLUE Score (VGS), to capture the efficacy and efficioncy of an FM on video understanding. However, VGS might be dominated by one or few datasets when becomes a serious issue, we should probably improve the score and/or retire the datasets from future versions of VideoGLUE. Indeed, VGS not perfect score that covers aspects of FMs in a comprehensive manner. For example, it does not account for FM's model size, model architecture, etc. We hope future rescarch will lead to new metrics to complement VGS and more comprehensive evaluation of FMs for visual tasks.
6 Conclusion
In this report, study four basee and three riseo-based foundation models and their adaptation capability on general vidco understanding tasks. Experiments are conducted on three hallmark video tasks, eight diverse datasets with four distinct adaption methods. Our study shows existing image-based FMs performs well on some appearance-rich video datasets, while rideo-based FMs tend to achieve better on motion and temporal reasoning. Four studied adaption methods curve different landscape, revealing the critical role of considering adaption methods as an organic part of evaluating FMs. Finally, we propose single metric VGS to represent the video task adaptation efliciency of FMs. We hope research provides useful resources for evaluating and analyzing video foundation models, and address the current gap in foundation model evaluation within the vidco domain.
7 Acknowledgement
We would like to thank Xuhui Jia and Sergey Toffe for reviewing and providing feedback on this paper. We thank Albert Shaw on their early investigation on the UNIT architecture. We also thank David Ross, Rahul Sukthankar, and Tomas Izo for their support and leadership on this project.
11

---

Published in Transactions on Machine Learning Research (10/2024)
Supplementary Materials
We first discuss the ethcial concerns and broader impact of this work (Section Aj. We detail the datasets (Sec- tion B
models (Section [). and training setups (Section D in the supplementary materials to improve this work's 8 reproducibility. Besides, Section includes more experimental studies to strengthen the main text.
A Ethical concern and broader impact
Ethical concern. We evaluate FMs on three video lasks, eight datasets in total. We select the tasks and datasets based on their popularity and representativeness. Although carefully designed, our benchmark inovitably inhcrited some ethical concerns from those datasets. For instance, many of the datasets are curated by crawling videos from the Internet, which do not proportionately represent the experiences of the global population and can potentially lead biased evaluations of FMs. Moreover, the video datasets involve human daily activities, leading to privacy concerns about the human actors in the videos. How to evaluate FMs for video understanding in fair and preeaccippeeeiingggg manner could be an important direction for future research.
Broader impact. Our research reveals the need and tremendous opportunities to research video-first FMs by improving pretraining video data and methodologies. Our studies on different adaptation methods on versatile tasks confirms that both tasks and adaptation methods matter when it comes to the evaluation of FMs, shodding light on the already vibrant area of FM adaptations. Finally, we hope our rescarch could inspire research on foundation models development and video understanding in general, along with their applications in the real world.
B
Video understanding datasets
Table 7: Summary of dataset publishing year, venue and citations (as of October 17, 2024).
Task
Dataset
Year
Vemue
Citation
Kinetics-100 Kay al.
2017
2017
arXiv
4,558
Moments Time Monfort cf al.
2019
2018
TPAMI
651
VC
Something-Someethng Something v2 Goyal et al.
2017
2017
ICCV
1,582
Diving48
Li al. 2018
2018
ECCV
369
Charades
Sigurdsson et al.
2016
2016
ECCV
1,415
TAL
ActivityNet v1.3 Fabian Caba Heilbron Niebles
2015
2015
CVPR
2,881
AVA v2.2 Guu
2018
2018
CVPR
1,000
STAL
AVA-Kinetics
[Li et al. 2020
2020
arXiy
151
In Table we show the publishing year, venues and citations demonstrate the popularity and community acceptance of datasets of our choice. Below we provide dataset details.
B.1 Appearance-focused action recognition
Video classification is task of classifying videos into pre-defined labels, with the major focus on human actions.
Kinetics-400 Kay et al. 2017 (K400) is a large-scale, high-quality video datasct widely used as a standard video classification benchmark. contains more than 250K video clips with annotations of 400 human daily actions. The actions are human focused and cover a broad range of classes including numan--uuma interactions and human-object interactions. Although the video clips span scconds on average, many studies (Sevilla- Lara et al.] 22021] Wang et al.) (2018b) have pointed out the task could be casily solved on the Kinetics datasets by inferring from the static objects appeared or background environment. motion information is
12

---

Published in Transactions on Machine Learning Research (10/2024)
less important than the visual appearance. Hence, we categorize Kinetics400 as an appearance-focused action classification dataset.
Moments in Time (Monfort et al., 2019) (MiT) is a large-scale video event classification dataset, with one million human annotated short video clips (around 3 seconds each). The temporal span corresponds to the averaged duration of human working memory and is a temporal envelope holding meaningful actions between people, objects, and phenomena. Videos in MiT are annotated with 339 most used verbs in the English vocabulary.
B.2 Motion-focused action recognition
Videos contain much more commionsense knowledge than still images do, such as an object's motion palierns and the causal consequences of an action, just to name a few. However, appearance- based benchmarks do not evaluate a model's understanding of such commonsense knowledge, complex scenes, and situations. In observance of this, some video datasets have been proposed and studied in recent years with the focus on motions and common-sensing reasoning that are prosperous in video data.
Bomething-Something v2 [Goyal et al. 2017) (SSv2) is a collection of around 200K videos of human performing pre-defined, basic actions with everyday objects. There are 174 unique labels in total depicting atomic hand manipulations, like putting something into something, turning something upside down or covering something with something. This dataset benchmarks model's fine-grained understanding capability of object motions and scene changes by making the label space atomic-action-fouuss and background-invarrien
Diving48 (Li et al. 2018) (D48) is introduced to evaluate model's 's dynamic reasoning capability. The video clips in this dataset are obtained by segmenting online vidcos of major diving competitions. In total, there are around 18K videos annotated with 48 classes. Because of its standardization, the diving scenario is purposefully chosen to avoid the scene, object, and person biases.
B.3 Multi-label daily action classification
Most of current action classification datascts involve video clips with a clean snapshot of a single action. In contrast, humans perform daily complex activities step-by-step, simultaneously, or in an interleaving manner. Towards more comprehensive human daily activity reasoning, Charades (Sigurdsson et al. 2016 is introduced. Different from web-collected datasots whose contents are more structured, Charades is collected by crowd-sourcing from hundreds of actors recording their vidcos in their own homes, acting out casual everyday activities. Charades brings in more diversity into the video classification task due to its close-to- daily-life setting. Its videos are 30 seconds long average and have multi-label annotations testing models' understanding of complex daily activities with multiple steps. Charades provides 110k vidcos with 137 action classes for training and evaluation.
B.4 Temporal action localization
Natural long videos contain scene changes and semantic shifts, while most of the existing video benchmarks formulate problems to focus on trimmed video clips. Such a gap introduces evaluation bias as clip-level benchmarks could not reflect a model's temporal feature discriminativeness, which is of key importance to solve long-form video understanding tasks. To comprehend the study on foundation models' video capabilities, we include the temporal action localization (TAL) task in our evaluation. The task of TAL to predict not only the action labels but also each action instance's temporal boundary in untrimmed vidcos. We adopt ActivityNet v1.3 (Fabian Caba Heilbron & Niebles 2015) as the dataset for the TAL task, which contains 10, 002 untrimmed videos in training and 4, 985 in validation. The video length in this dataset is between 5-10 minutes. In total, there are 200 types of activities annotated.
B.5 Spatiotemporal action localization
Spatiotemporal Action Localization (STAL) is a person-centric task that asks a system to localize actors and predict their atomic actions (Barker & Wright 1955 Gu et al. 2018 in transitory duration.
13

---

Published in Transactions on Machine Learning Research (10/2024)
Quary Tokan
Token
MLP
MLP
MLP
Norm
Norm
Mult-Head Sett Attontioo
Mubb-Hoad Self-Altention
Mulli-Hodd Self-Altetlon
Norm
Norm
Norm
Layer N
Layer N-k+1
Layer N
(a)
(b)
Figure 4: (a) Single-layer pooler head and (b) multi-layer attention pooling head for video classification and spatiotemporal action localization.
Table 8: Farly vs. late fusion image-native FMs. In this experiment, the frozen feature with single-layer pooler head is used.
K400
SSv2
Models
Early
Late
Early
Late
CLIP
70.5
75.2
38.1
41.0
FLAVA
67.9
71.3
40.4
40.6
CoCa
72.7
61.4
41.5
33.3
In AVA Gu et al. 2018). 15 minutes long movie clips are densely annotated at 1Hz. In the key frames, every person is localized using a bounding box and labels corresponding to actions being performed by the actor. The label vocabulary consists of 80 different atomic visual actions. There are 430 different movies in total.
AVA-Kinetics [Li et al. 2020 follows the same labeling protocol AVA, while its data source comes from the Kinetics700 (Kay et al. 2017) video pool. The dataset contains over 230K clips annotated with the 80 AVA action classes for each of the humans in key frames.
C Model details
C.1 Task head architectures
In Figure 4 we plot the task heads used in our video classification and spatiotemporal action localization experiments, namely, the simple pooler head and multi-layer attention pooling head. For temporal localization, please refer to Xu et al. (2020) for the task head's detailed architecture.
Figure illustrates the encoder adapter layer's architecture. In the the adapter layer, only the down-sample layer, up-sample layer, and the scaling factor tunable.
C.2 Image-to-vided adaptation
Adapting image backbones to vidco tasks requires us to fuse the image embeddings at some point in the network and also introduce additional temporal information.
14

---

Published in Transactions on Machine Learning Research (10/2024)
UpFFN
Adapter
*
Narm
DownFFN
Mult--Had
Self-Altention
Norm
The adapter used in vision transformer. In the adapter layer, only the down-sample layer, up-sample layer, and the scaling factor are tunable. Between the down-sample layer and up-sample layer, activation function is applied, which in our case is ReLU.
Figure 5:
Table 9: Ablation study on the temporal positional embodding for image-to-vidco adaption. We choose FLAVA Singh el al. 2022 with the frozen feature selting this experiment.
Adds temporal
VC-A
VC-M
VC-ML
positional embedding?
K400
MiT
SSv2
D48
Charades
x
71.3
29.7
30.3
41.6
10.7
71.3
29.7
40.6
45.9
12.6
We consider two choices, early-fusion and late-fusion, and ablate them in the frozen feature setting in Table g In both carly-fusion and lato-fusion, we first apply the projection layer on each frame independently to embed pixel patches into embedding tokens. We then average-pool the embedding tokens from nearby frames reduce the sequence length n w. In the early-fusion setting, we pass all tokens together to the image backbone to extract video features. In late-fusion, we pass each set of h w tokens independently to the image backbone. Empirically, find that the FLAVA (iighh et al. 2022 and CLIP (Radford et al. 2021 models do better with late-fusion while CoCa. YYu et al. 2022 does better with early-fusion.
Furthermore, we ablate the importance of temporal information using the frozen-features from FLAVA Singh et al. 2022). In Table 9. we find that adding temporal positional embedding to the input is essential for D48 [i et al. 2018), SSv2 (Goyal al. 2017), and Charades (Sigurdsson et al.) (016)) while necessary
for K400 (aay et al. 2017) and MiT (Monfort al. 2019 This supports our grouping that K400 and MiT are appearance-foouseed datasets.
Based on these findings, use late-fusion for FLAVA (Singh et al. 2022 and CLIP (Radford et al. 2021 and carly-fusion for CoCa Yuu et al. 2022 We add Icarnable temporal positional embeddings for all the image-native FMs.
D Task-specific hyperparameters
In the following, we provide experiment settings and hyperparamters we used in this study. Table I0 we list the hyperparameters we applied the video classification task. In Table [1 we present the hyperparametens we used on spatiotemporal action localization. In Table 12 we present the hyperparameters we used on
temporal action localization task.
15

---

Published in Transactions on Machine Learning Research (10/2024)
Table 10: Experimental configurations for video classification tasks. We let learning rate and weight decay to be tunable per model allow some flexibility for task adaptations.
Configurations
K400
MiT
SSv2
D48
Charades
General
Batch size
256
256
256
256
256
Training epochs
150
50
50
100
50
VIT sequence length
8 x
14
14
8 x 14
14
8
x
14
14
8
Optimization
Optimizer
AdamW
AdamW
AdamW 0.9
AdamW
AdamW
Optimizer momentum
0.9
0.9
0.9
0.9
Learning rate schedule
Cosine decay
Cosine decay 5%
Cosine decay
Cosine decay
Cosine decay
Warmup ratio
5%
5%
5%
5%
Data augmentations
Random horizontal flip
Yes
Yes
No
Yes
No
Aspect ratio
(0.5, 2.0)
(0.5, 2.0)
(0.5, 2.0)
(0.5, 2.0)
(0.5, 2.0)
Area ratio
(0.3, 1.0)
(0.3, 1.0)
(0.3, 1.0)
(0.3, 1.0)
(0.3, 1.0)
RandAug
(9, 0.5)
(9, 0.5)
MixUp
0.8
0.8
CutMix
1.0
1.0
Evaluation
Multi-clips
4
4
Multi-views
3
3
3
Segment-based sampling
No
No
Yes
No
No
We performed greedy search the learning rate and weight decay all our experiments while keeping most other hyperparameters (c.g., data augmentation magnitude, dropout rate, drop path rate, etc.) consistent across different models and datasets. Specifically, we start with learning rate le-4 and weight decay 1e-5 and uniformly sample learning rates and weight decay factors with a rate of 5 and 10, respectively, centered around the starting points. After the first round, we pick the best-identified learning rate and weight decay factor as the new starting point and conduct another round sampling with a rate of We repeat another two three rounds of hyperparameter search (with rate 2) until the model's performance converges. This process is a tradc-off between computation costs and thoroughly examining an FM's performance under each experiment setup. The search ranges for the learning rate and weight decay are [4e-5, 2.5e-3) and [le-6, le-4], respectively. We found that the learning rate is the most crucial factor when adapting FM to downstream video understanding tasks.
E
More studies
E.1 Large model adaptations
For the completeness of this report and reader's reference, in Table 13 we report experimental results under our settings with large FMs under the frozen backbone with one pooler head setup.
VidcoMAE v2-B B/BDD WWag et al. 2023) denotes the VIT-B model distilled from ViT-g on the Kinctics710
datasets f MeleoMAE-v2-g (Wang al.
2023) the model that pretrained on UnlabeledHHybrid dataset,
while VideoMAE-v2--g/5
(Wang et al.
2023 conducts further finetuning using supervised training
Kinetics710. InternVideo-v2-p (Wang et al. 2024h and VideoPrism-g (Zhao et al.] 2024) are two video foundation models with multi-stage pre-training on curated in-house web video data. For InternVicoo-v2gg. we use their stage-2 checkpoin For videoPrism-gg, we use their final checkpoint.
16

---

Published in Transactions on Machine Learning Research (10/2024)
Table 11: Experimental configurations for spatiotemporal action localization.
Configurationss
AVA v2.2
AVA-Kineties
General
Batch size
256
256
Training epochs
50
50
VIT sequence length
8 x 16 x 16
x 16
Optimization
Optimizer
AdamW
AdamW
Optimizer momentum
0.9
0.9
Layer decay
0.75
0.75
Learning rate schedule
Cosine decay
Cosine decay
Warmup ratio
5%
5%
Data augmentations
Random horizontal flip
Yes
Yes
Random scale
(0.5, 2.0)
(0.5, 2.0)
Random color augmentation
Yes
Yes
Table 12: Experimental configurations for temporal action localization.
Configurations
ActivityNet v1.3
General
Batch size
32
Training epochs
10
Feature extraction
Frame rate (FPS)
15
Per-clip length (second)
16
Clip stride
16
Optimization
Optimizer
AdamW
Optimizer momentum
0.9
Learning rate schedule
Cosine decay
Table 13:
Evaluating large-scale FMs when using frozen feature with a one-layer pooler head. We report the Top-1 accuracy on K400, MiT, D48, SSv2 and MAP on Charades.
VC-A
VC-M
VC-ML Charades
Models
K400
MIT
SSv2
D48
InternVideo-L
78.6
33.7
67.4
69.6
20.9
VideoMAEEvv-BBDDL
86.7
38.9
57.7
61.4
33.2
VideoMAE-v2-g
59.7
20.7
44.2
42.5
12.7
VideoMAA-v2-g/FI
82.1
35.0
56.1
60.5
22.4
InternVideo-v2-g
85.0
43.0
61.6
53.1
40.9
VideoPrism-gg
86.6
44.7
67.4
66.1
61.0
17

---

Published in Transactions on Machine Learning Research (10/2024)
Table 14: Benchmark FMs adaptation on video understanding tasks under sample-eficient transfer learning. This table shows Top-1 classification accuracy and the relative accuracy training with 100% data (shown in parentheses). Results are achieved by using frozen features with pooler head.
K400
SSv2
Models
1%
10%
100%
1%
10%
100%
CLIP
36.9 (46.2%)
66.8 (83.6%)
79.0
8.7 (19.3%)
25.1 (55.5%)
45.3
FLAVA
14.4 (20.2%)
35.8 (50.3%)
71.3
7.2 (17.7%)
14.3 (35.3%)
40,6
CoCa
27.1 (37.8%)
48.9 (67.0%)
73.1
5.6 (13.4%)
20.9 (50.4%)
41.5
VATT
34.1 (45.4%)
63.7 (84.8%)
75.1
12.9 (22.4%)
37.6 (65.0%)
57.8
VideoMAE
15.5 (23.9%)
32.0 (49.2%)
65.0
13.7 (25.4%)
30.3 (56.2%)
53.9
InternVideo
20.4 (29.5%)
50.2 (72.4%)
69.3
19.5 (33.6%)
41.1 (70.7%)
58.2
E.2 Sample-efficient transfer learning
A strong FM should be able to adapt to downstream tasks with few training samples. In this section, we test the adaption ability of FMs in a sample-efficiet transfer learning setting. Particularly, we freeze backbones and train a pooler head to adapt the FMs on K400 and SSv2. For either dataset, we sample 1% and 10% data from the training set uniformly for training and evaluate on the full evaluation dataset.
We show our experimental results Table 14 To better understand the data efficiency, we also show the relative Top-1 accuracy for each model (shown in the bracket), which is defined the ratio between accuracy with fewer training examples and the accuracy achieved using all the training data. A higher relative Top-1 accuracy means the performance of the model is closer to its "full" capacity under the sample-effccient setting. We notice that the best performed model on each dataset in fully fine-tuned model also performs best in the few-shot setting. Especially, CLIP (Radford et al. 2021) achieves 46.2% and 83.6% relative Top-1 accuracy on K400 using only 1% and 10% of the training data, respectively. On SSv2, InternVideo (Wang et al.) 2022b achieves 33.6% and 70.6% relative Top-1 accuracy with only 1% and 10% of the training data.
18

---

Published in Transactions on Machine Learning Research (10/2024)
References
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. VATT: Transformers for multimodal self-supervised learning from raw video, audio and text. In NeurlPS 2021.
Maan-Bapistte Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning, In Newr/PS, 2022.
Humam Alwassel, Silvio Giancola, and Bernard Chanem. TSP: Temporally-semiitive pretraining of video encoders for localization tasks. In ICCV, 2021.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Diiitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. PaLM technical report. orXio preprint Xrv:22305 10403, 2023.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEIT: BERT pre-training of image transformers. ar preprint orXiv:2106.c 08254, 2021.
Roger G Barker and Herbert F Wright. Midwest and its children: The psychological ecology of an American town. Marriage and Family Living, 1955.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0 07258, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. In NewrIPS, 2020.
Shyamal Buch, Cristobal Eyzaguire, Adrien Gaidon, Jiajun Wu, Fei-Fei, and Juan Carlos Niebles. Revisiting the "video" in video-language understanding. In CVPR, 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
Xi Chen, Xiao Wang, Soravil Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycuer, Basil Mustafa, Lucas Beyer, et al. PaLI: A jointly-scaled multilingual language-inage model. ar preprint Xio:2220 06709, 2022.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarlen Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. PaLM: Scaling language modeling with pathways. arxiv preprint arXiv:2204.022 1, 2022.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXi::1810.04880 2018.
Li Dong, Nan Yang, Wenhui Wang, Furn Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. Unified language model pre-training for natural language understanding and generation. In NewrIPS, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. ariii preprini 1rXio:2010.1110 11929, 2020.
Bernard Ghanem Fabian Caba Heilbron, Victor Escorcia and Juan Carlos Niebles. ActivityNet: large-scale video benchmark for human activity understanding. In CVPR, 2015.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In ICCV, 2021.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. SlowFast networks for video recognition. In ICCV, 2019.
Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. In CVPR, 2021.
19

---

Published in Transactions on Machine Learning Research (10/2024)
Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked auloencoders as spaliotemporal learners. arXiv preprind Siv:2205.09113 2022.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Stueller-Freitaae et al. The "something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
Chunhui Gu, Chen Sun, David A Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijyanarraimmhn, George Toderici, Susanna Rioco, Rahul Sukthankar, et al. AVA: A video dataset of spatio-tepporally localized atomic visual actions. In CVPR, 2018.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. ICLR, 2021.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lx, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. ar preprint urXie:2302. 14045, 2023.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, et al. Spatial transformer networks. In NewrIPS, 2015.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual- langual models for efficient video understanding. In ECCV, 2022.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The Kinetics human action video dataset. arXiv preprint arXiv: 765506950, 2017.
Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong. MoviNets: Mobile video networks for efficient video recognition. In CVPR, 2021.
Jie Lei, Tamara L. Berg, and Mohit Bansal. Revealing single frame bias for andeo-and-oo language learning. In ACL, 2023.
Ang Li, Meghana Thotakuri, David A Ross, João Carreira, Alexander Vostrikov, and Andrew Zisserman. The AVA-Kinetics localized human actions video dataset. erXiv preprint 200,00, 2020.
Chunyuan Li, Haotian Liu, Liunian Harold Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Yong Jae Lee, Houdong Hu, Zicheng Liu, et al. ELEVATER: A benchmark and toolkit for evaluating LanguageAnggnented Visual Models. arXiv preprint 2rXiv:200400870 08790,, 2022a.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang, and Yu Qiao. UniformerV2: Spatiotemporal learning by arming image ViTs with video UniFormer. arXie preprint arXis:2211.09552 2022b.
Yingwei Li, Yi Li, and Nuno Vasconcelos. Resound: Towards action recognition without representation bias. In BCCV, 2018.
Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang. Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. In ECCV, 2022.
Xiaolong Liu, Song Bai, and Xiang Bai. An empirical study of end-to-end temporal action detection. CVPR, 2022.
Mathew Monfort, Alex Andonian, Bolei Zhon, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, understanding. JEEE TPAMI, pp. 1-8, 2019.
Moments Time dataset: One million videos for event
OpenAT. GPT-4 Technical Report. http://dwooooocooooooo/oop/p/oa/
2022.
20

---

Published in Transactions on Machine Learning Research (10/2024)
Maxime Oquab, Timothée Darcel, Théo Moutakanni, Huy Vo, Marc Szafraniee, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual fealures without supervision. In TMLR, 2023.
Viorica Patrancean, Lucas Smaira, Ankush Gupta, Adria Recasens, Larisa Markeeva, Dylan Banarse, Skanda Koppula, Mateusz Malinowski, Yi Yang, Carl Doersch, et al. Perception Test: A diagnostic benchmark for multimodal video models. In NewrlPS, 2024.
AJ Piergiovanni, Weicheng Kuo, and Anelia Angelova. Rethinking video VTIs: Sparse video tubes for joint image and video learning. In CVPR, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela. Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. ICML, 2021.
Jathushan Rajasegaran, Georgios Pavlakos, Angion Kanazawa, Christoph Feichtenhofer, and Jitendra Malik. On the benefits of 3D pose and tracking for human action recognition. In CVPR, 2023.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and llya Sutskever. Zero--hot text-to- toxt-to-image generation. In ICML, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards real-time object detection with region proposal networks. In NewriPS, 2015.
Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, et al. Scaling up models and data with t5x and seqio. arXiv preprint arXiv:2203.1 17189, 2022.
Laura Sevilla-Lara, Shengxin Zha, Zhicheng Yan, Vedanuj Goswami, Matt Feiszli, and Lorenzo Torresani. Only time can tell: Discovering temporal data for temporal modeling. In WACV, 2021.
Gunnar A Sigurdsson, Güil Varol, Xiaolong Wang, Ali Farhadi, Ivan Laptev, and Abhinav Gupta. Hollywood in Homes: Crowdsourcing data collection for activity understanding. ECCV, 2016.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Mareus Rohrbach, and Douwe Kiela. FLAVA: A foundational language and vision alignment model. In CVPR, 2022.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. VideoMAE: Masked autoencoders are data-efficient learners for self-supervised video pre-training. arXiv preprind arxiv:2203.122 12600, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jacob Uszkoreit, Llion Jones, Aidan N. Comez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint ar
2018a.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32, 2019.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dalua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks for action recognition videos. IEEE TPAMI, 41(11):2740-277 2755, 2018b.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. VideoMAE v2: Scaling video masked autoencoders with dual masking. In CVPR, 2023.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammedd, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: BEIT protraining for all vision and vasion-language tasks. aix preprint arxio:2208. 10442, 2022a.
Xiang Wang, Zhiwu Qing, Ziyuan Huang, Yutong Feng, Shiwei Zhang, Jianwen Jiang, Mingqian Tang, Changxin Gao, and Nong Sang. Proposal relation network for temporal action detection. arXin preprint Xrxi0:106.. 1 11812, 2021.
21

---

Published in Transactions on Machine Learning Research (10/2024)
Yi Wang, Kunchang Li, Yizhuo Li, Yinan He, Bingkun Huang, Zhiyu Zhao, Hongjie Zhang, Jilan Xu, Yi Liu, Zun Wang, et al. InteraVideo: General video loundation models via generalive and discriminative learning, ar preprint 2r1::222 03191, 2022b.
Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. InternVide02: Scaling video foundation models for multimodal video understanding. ar preprint 2rXiv:2403.1500 15377, 2024.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models. arXiv preprint arXi::2206.0700 2022.
Mengmeng Xu, Chen Zhao, David S Rojas, Ali Thabet, and Bernard Ghanem. G-TAD: Sub-graph localization for temporal action detection. In CVPR, 2020.
Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui VideoCoCa: Video-text modeling with zero-shot transfer from contrastive captioners. orXiv preprint erXiv:2212 04979, 2022.
Taojiannan Yang, Yi Zhu, Yusheng Xie, Aston Zhang, Chen Chen, and Mu Li. ATM: Adapting image models for efficient video action recognition. arXiv preprint or #xXi:230200302, 2023.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojiaba Seyedhosseini, and Yonghui Wu. CoCa: Contrastive captioners image-text foundation models. arXie preprint S.i::2205501111, 2022.
Cie Zhang, Jianxin Wu, and Yin Li. Actionformer: Localizing moments of actions with transformers. In BCCV, 2022.
Long Zhao, Nitesh B Gundavarapu, Liangzhe Yuan, Hao Zhou, Shen Yan, Jennifer J Sun, Luke Friedman, Rui Qian, Tobias Weyand, Yue Zhao, et al. VideoPrism: A foundational visual encoder for video understanding. In ICML, 2024.
22


Reviews:
[{"review_id": "aWa0K7zXeU", "reviewer": "Reviewer_Twzc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The study includes diverse tasks and datasets.\n- It presents a detailed experiment protocol for training and adaptation methods.\n- The supplementary material provides all hyperparameters, suggesting that the experiments could be well replicated.\n- The evaluation considers cost.\n\nWeaknesses:\n- There is an imbalance in the scores averaged for the VGS, making it less informative than the tables themselves. A score in relation to the state-of-the-art (task-specialized models) might be more informative.\n- The sudden appearance of DINO-v2 is unexplained and not considered in the conclusion. It is also missing from essential figures such as Figure 1 and Figure 3 (right).\n- It is very difficult to get an intuition about what the VGS actually does. \n- Overall limited methodological advancements: While handling FMs and their task adaption has become an important aspect in (resources-aware) deep learning research, I think that the methodological advancements in this paper are rather limited, since the major part of the contributions are evaluations. Hence, I consider the potential impact of this paper as limited as well and do not see how it would advance the field.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- The authors evaluate the generalization capabilities of existing foundation models (FMs) in video understanding and cost across eight prominent datasets, encompassing action recognition, temporal, and spatiotemporal localization tasks.\n- They implemented four adaptation methods for fine-tuning the FMs. VideoGLUE revealed that task-specialized models generally outperform FMs. \n-Video-native FMs typically surpass image-native FMs in motion-focused and multi-labeled video classification (VC) as well as temporal action localization tasks. \n- Additionally, video-native FMs adapt better with light adaptations, while image-native FMs excel with full end-to-end finetuning.\n- Furthermore, they introduce a novel VideoGLUE Score (VGS) representing the overall efficacy and efficiency of the FMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Changes\n- DINO-v2 missing from essential figures: Figure 1 and 3.\n\nQuestions\n- It is quite unexpected that the image-native FMs win in full end-to-end fine tuning compared to video-native FMs. What are possible reasons here?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZF4V7nIf8Y", "reviewer": "Reviewer_HnNq", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "For context, I would like to preface my review in that I am no expert in the particular fields of computer vision or video understanding but have general awareness of a set of foundation models present in the field as well as general ML expertise.\n\n## Strengths\nOverall, I consider this a strong piece of work that fits well with the audience of TMLR. The field of video understanding is relevant for many research directions and applications but also understudied compared to typical image-based computer vision problems. Therefore, I believe that a standardised and well defined evaluation protocol for this field can be a meaningful contribution. \n\nIn particular, I appreciate the clear justification and description of the evaluation protocol as well as the consideration of varying adaptation methods as well as studied models and datasets. Furthermore, I would like to positively highlight the consideration of training, inference, and memory cost which is highly valuable information for any practitioner.\n\n## Weakness\n1. Table 3 and Table 6 that list evaluation results for end-to-end finetuning and low-rank adaptation are missing results for the ANet dataset. This is even more confusing given that Section 4.1 is actively discussing results in the ANet dataset. Any clarification would be greatly appreciated.\n2. Among image-native FMs, the study solely focuses on image-text contrastive trained models. I agree that these are prominent representatives for image-native FMs that have been shown to perform well on a range of tasks, but it still is surprising to me that no other category of image-native FM are considered. In its current form, any claims of the work about \"image-native FMs\" as a whole seem to have only weak support and should have the strong asterisk that these findings were only based on image-text pre-trained FMs. In particular, the following categories of models would provide useful additional comparison points for the study:\n\t- Non-text based self-supervised pre-trained models, e.g. DINOv2 [1]\n\t- Segmentation pre-trained models, e.g. SAM [2]\n\t- Object detection models, e.g. GroundingDINO [3] and Co-DETR [4]\n\t- Classification trained models, e.g. trained on ImageNet\n3. The paper makes few claims/ statements that do not appear to be fully supported by the provided evaluation results.\n\t1. In 4.1, the work states that \"all six FMs significantly underperform the task-specialized models on the video tasks at the comparable model scale, indicating the lack of strong video-focused FMs\" -- this seems to hold for most but not all datasets. For the MiT and SSv2 datasets the best fine-tuned models seem comparable to the best specialised ones. The language here should be relaxed since the claim is not fully supported by the provided data.\n\t2. In 4.1, the work states \"video-native FMs outperform image-native FMs on SSV2, Charades, and ANet\". However, the image-native CoCa model is the 2nd best overall model and only behind one video-native model in both SSv2 and Charades, outperforming several of the video-native models on both datasets. Again, the claim that video-native FMs perform image-native FMs on these datasets does not seem to be fully supported by the data and should be relaxed. Also, it appears odd how the sentence makes any claims about ANet while the data for ANet appears to be missing in Table 3 (see weakness 1.).\n\t3. Similar to the previous claim, at the end of Section 4.1, the paper hypotheses that CoCa does not perform as well on the SSv2, Charades, and ANet datasets \"[...] probably because CoCa, pretrained using image-text pairs, does not capture sufficient motion signals required for understanding SSv2\". However, CoCa is barely performing worse than the best-performing model (InternVideo) on the SSv2 dataset according to Table 3, and clearly outperforms both other video-native models. Thus, this claim about the suggested failure of CoCa seems surprising and against provided data.\n4. What are the model sizes of the studied FMs? Are these models comparable in terms of number of parameters and pre-training compute budget? It would be useful if Table 1 could provide indications of the model size as well as the pre-training dataset/ compute budget to provide context for the following analysis.\n5. Figure 3 (left) specifies the inference cost, training cost, and number of parameters for each adaptation method. This confuses me somewhat as I would have expected these metrics to depend on the FM that is being adapted. In particular the inference cost and training cost of the E2E would strongly depend on the size of the underlying FM. Are these metrics reported for a particular FM? If so which, and if not, why do these metrics not depend on the FM as I would have expected? Any clarification would be appreciated. Also, related to Weakness 4. and Figure 3 (right), which reports the VideoGLUE scores normalised by cost, it would be useful to know the cost of each evaluated FM model to better interpret the VideoGLUE scores.\n6. As stated in Section 4.2, the ranking of FMs heavily depends on the type of adaptation. In particular, \"image-native FMs outperform video-native FMs under end-to-end finetuning and the adapter, but it becomes the inverse in the other two adaptation methods.\" (from last paragraph in 4.2). However, this nuance is lost in the suggested VideoGLUE score which, as far as I understand, represents the aggregated performance of a FM while considering all four adaptations studied in this work normalised by the training/ inference/ memory cost. Providing separate scores normalised by cost for each of the four adaptation methods would help to provide indications of which model performs best under a particular adaptation method. This is particularly useful since practitioners will want to know which model performs best under its most suitable adaptation method (that fits their computation budget) rather than which model performs best across a wide range of adaptation methods.\n\t1. Following this idea and usecase, it would be useful if the work could provide a comparison of each FM under the best-performing adaptation method for that particular model, likely excluding end-to-end finetuning which usually performs best but can often be too costly to do (as also shown in Figure 3). This would provide useful indication for practitioners of 1. which model should be finetuned with which adaptation technique and 2. what performance can be expected of each model given its most suitable finetuning approach.\n7. Figure 1 appears to summarise some results but the caption and figure do not make it clear which models are compared here, in particular it is unclear which adaptation protocol was followed. From the numbers, I am assuming that Figure 1 summarises the results for the end-to-end finetuning experiment also shown in Table 3 in which case the caption should state that these results were obtained after end-to-end finetuning of all FMs.\n\nAs a sidenote, I would like to highlight that the authors mention that they intend to release open-source code for the benchmark upon acceptance. I believe that a thorough and well-documented release of this code would be a very strong addition to the paper, simplifying future work to evaluate novel models following the proposed benchmark suite and methodology for fair comparisons.\n\n--- \n\n## References\n[1] Oquab, Maxime, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez et al. \"Dinov2: Learning robust visual features without supervision.\" arXiv preprint arXiv:2304.07193 (2023).\n\n[2] Kirillov, Alexander, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao et al. \"Segment anything.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026. 2023.\n\n[3] Liu, Shilong, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li et al. \"Grounding dino: Marrying dino with grounded pre-training for open-set object detection.\" arXiv preprint arXiv:2303.05499 (2023).\n\n[4] Zong, Zhuofan, Guanglu Song, and Yu Liu. \"Detrs with collaborative hybrid assignments training.\" In Proceedings of the IEEE/CVF international conference on computer vision, pp. 6748-6758. 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The work proposes a evaluation protocol and benchmark called VideoGLUE which is designed to evaluate the capabilities and adaptability of foundation models for video understanding. The benchmark consists of eight datasets and considers four methods of finetuning the evaluated models for the respective downstream task/ dataset: end-to-end finetuning, only training a single attention layer on top of the output features of the frozen model, training multiple attention layers that operate on the output features of the last few layers of the frozen model, and low-rank adaptation. The work follows their proposed evaluation protocol and benchmark to evaluate three image-native foundation models as well as there video-native foundation models. Findings suggest that in particular the image-native CoCa model is well suited for the majority of video understanding tasks but few tasks see video-native models perform best. The work also compares the suitability of the considered adaptation methods for the different studied models and analysis the training, inference, and memory cost associated with each adaptation method, and proposes a metric which represents the average performance of each model normalised by its induced cost.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "## Major/ Critical Changes\n1. Add results for the ANet dataset missing in Table 3 and Table 6. (see Weakness 1.)\n2. Add more different types of image-native FMs in the evaluation, or relax any statements made about \"image-native FMs\" as a whole and provide a justification for the selection of image-native FMs. (see Weakness 2.)\n3.  Relax and/ or correct not sufficiently supported claims and statements made in Section 4.1. (see Weakness 3.) \n\n## Minor Changes\n4. Provide information on the number of parameters and the pre-training compute budget of studied FMs in Table 1 or elsewhere. (see Weakness 4.)\n5. Clarify why the training, inference, and memory cost of adaptation methods does not depend on the underlying FM, or state for which FM Figure 3 (left) is reporting results for. (see Weakness 5.)\n6. Clearly state for which adaptation method Figure 1 is presenting results for. (see Weakness 7.)\n\n## Further Suggestions (Minor)\nProvide separate VideoGLUE-like scores for each of the four adaptation methods to provide indication of which model performs best under a particular adaptation method. (see Weakness 6.)\n\nAlternatively/ Additionally, one could provide a comparison of each FM under the best-performing adaptation method for that particular model, likely excluding end-to-end finetuning which usually performs best but can often be too costly to do. (see Weakness 6.1)", "broader_impact_concerns": "I do not foresee any ethical implications that are particular to this work and require discussion.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "isvKN1Y4Us", "reviewer": "Reviewer_LvKA", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well written and easy to follow. The visualization is good.\n\n2. This work addresses an important gap by introducing the first benchmark focused on evaluating FMs for video understanding.\n\n3. The introduction of the VideoGLUE protocol, encompassing diverse tasks and datasets, provides a robust framework for evaluating FMs in video understanding.\n\nWeakness:\n1. While the paper makes a good case for focusing the video understanding evaluation on action-related tasks, there are other important aspects of video understanding not covered here, such as scene classification, object tracking, video question answering, etc. The scope is somewhat narrow.\n\n2. Unlike some foundation model evaluation efforts that provide human performance on the tasks for comparison, this paper does not include any human baselines. This makes it hard to contextualize how far both the task-specialized models and foundation models are from human-level video understanding.\n\n3. The paper mentions examining the \"efficiency\" of the foundation models but does not actually report any metrics related to computational cost, such as FLOPs, parameters, latency etc. This is an important practical consideration.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes VideoGLUE, a benchmark for evaluating the video understanding capabilities of existing foundation models (FMs). The authors design an experiment protocol consisting of three main tasks (action recognition, temporal localization, and spatiotemporal localization), eight popular datasets, and four adaptation methods to tailor an FM for downstream tasks. They evaluate six visual and multimodal FMs and jointly profile their efficacy and efficiency using various cost measurements under different scenarios. The evaluation result reveals that task-specialized models outperform them, video-native FMs generally perform better than image-native FMs on motion-rich videos and multi-action videos, and adaptation methods significantly impact the performance of FMs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. While the paper draws conclusions about video-native models outperforming image-native models on certain tasks, it does not dig deeper to analyze what architectural choices or pretraining approaches lead to better video understanding in the foundation models. A more thorough ablation study could yield useful insights.\n\n2. The datasets are described as \"well-received by the research community\" but the selection criteria are not fully specified. It's possible the particular datasets chosen could bias the results. To strengthen the validity of the findings, the authors should clearly specify the dataset selection criteria and justify why these datasets are representative of the broader landscape of video understanding tasks.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission received three expert reviews; after discussions, two reviewers recommended acceptance, and one disappeared despite a few reminders. Reviewers appreciated the thorough evaluation and agreed that the claims are well supported; they also thought the discussion was helpful in addressing their concerns.  The AE agreed with the reviewers and recommended acceptance.
Decision: Accept (poster)