ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning
Authors: Junting Pan
Ziyi Lin
Xiatian Zhu
Jing Shao
Hongsheng Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: parameter-efficient transfer learning
video recognition
adapters

Abstract: Capitalizing on large pre-trained models for various downstream tasks of interest have recently emerged with promising performance. Due to the ever-growing model size, the standard full fine-tuning based task adaptation strategy becomes prohibitively costly in terms of model training and storage. This has led to a new research direction in parameter-efficient transfer learning. However, existing attempts typically focus on downstream tasks from the same modality (e.g., image understanding) of the pre-trained model. This creates a limit because in some specific modalities, (e.g., video understanding) such a strong pre-trained model with sufficient knowledge is less or not available. In this work, we investigate such a novel cross-modality transfer learning setting, namely parameter-efficient image-to-video transfer learning. To solve this problem, we propose a new Spatio-Temporal Adapter (ST-Adapter) for parameter-efficient fine-tuning per video task. With a built-in spatio-temporal reasoning capability in a compact design, ST-Adapter enables a pre-trained image model without temporal knowledge to reason about dynamic video content at a small ~8% per-task parameter cost, requiring approximately 20 times fewer updated parameters compared to previous work. Extensive experiments on video action recognition tasks show that our ST-Adapter can match or even outperform the strong full fine-tuning strategy and state-of-the-art video models, whilst enjoying the advantage of parameter efficiency.

ST-Adapter: Parameter-Efficiet Image-to-Video Transfer Learning
Junting Pan'r, Ziyi Lin "n,",, Xiatian Zhu?, Jing Shao', Hongsheng Li
Multimedia Laboratory, The Chinese University of Hong Kong 2 Surrey Institute for People-Centred Artificial Intelligence, CVSSP, University of Surrey Centre for Perceptual and Interactive Intelligence Limited
Abstract
Capitalizing on large pre-trained models for various downstream tasks of interest have recently emerged with promising performance. Due to the ever-growing model size, the standard full fine-tuning based task adaptation strategy becomes prohibitively costly in terms of model training and storage. This has led to a new research direction in parameter-efficient transfer learning. However, existing attempts typically focus downstream tasks from the understanding) of the pre-trained model. This creates a limit because some specific modalities, (e.g., video understanding) such strong pre-trained model with sufficient knowledge is less or not available. In this work, we investigate such novel cross-modality transfer learning setting, namely parameter-efficent image-to-video transfer learning. To solve this problem, we propose new Spatio- Temporal Adapter (ST-Adapter) for parameter-efficient fine-tuning per video task. With built-in spatio-temporal reasoning capability in compact design, ST- Adapter enables a pre-trained image model without temporal knowledge reason about dynamic video content at small (~88% per-task parameter cost, requiring approximately 20 times fewer updated parameters compared to previous work. Extensive experiments on video action recognition tasks show that our ST-Adapter can match or even outperform the strong full fine-tuning strategy and state-of-the- art video models, whilst enjoying the advantage of parameter efficiency. Code and model are available at lhttps: //github. www httm//inziii99//// stt
modality (e.g., image
Introduction
In the NLP field, almost all the state-of-arts across wide range of downstream tasks have been achieved by adapting from large pretrained models (a. foundation models (7) such as BERT [I5] and GPT [54 8) The de facto standard approach to adapting a pretrained model to down-stream tasks is fine-tuning either fully or partially (e.g., linear probing by training the newly added multi-layer perceptron layers on the top alone), subject to the condition of adopting a similar network architecture the pretrained model. Nonetheless, given increasingly larger whilst models (e.g., GPT-3 with 175B parameters), fully fine-tuning the whole model for every single downstream task would become prohibitively expensive and infeasible in terms of training cost and model storage. This could significantly restrict their deployment and usability in real-world applications. In this context, series of NL.P works has been introduced towards efficient transfer learning with better trade-offs between parameter and accuracy [25] 24
stronger foundation
36
This trend has recently motivated the computer vision community. For example, the CLIP model trained with 400 million web image-text pairs, achieves promising performances on a variety of
55
"Equal contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Add & Norm
Add & Norm
Spatio-Temporal Adapter
Feed Forward
Feed Forward
Upsample
Add & Norm
Add Norm
DW 3D-Conv
Multi-Head Self-Attention
Multi-Head Self-Attention
Downsample
Temporal Self-Att Belf-Attention Layers
ST ST-Adapter
Layers
(a) Full Fine-tuning [6]
(b) ST-Adapter Fine-tuning
Action Recognition Acc: 81.7
Action Recognition Acc: 82.0
Fixed during training
Updated Param: 141.2%
Updated Param: 8.3%
Newly added to ViT [18]
Figure 1: Image-to-video transfer learning strategies. (a) The state-of- ofate-of-the-are methods for adapting
a pre-trained image model (e.g., ViT [16] in this example) to video tasks (e.g., action recognition)
usually adopt the paradigm of first designing temporal learning module and then fine-tuning the
whole network fully 22 6 9 This is parameter-inefficien since a specific instance of such a large model is resulted for each downstream task. In contrast, (b) we propose to only train a lightweight Temporal Adapter with much fewer parameters for each individual downstream task at a significantly smaller computational cost. Surprisingly, our method can match even surpasses the full fine-tuning based methods (including prior art video models in terms of accuracy), whist enjoying higher parameter efficiency and cheaper training cost.
image recognition and generation tasks. In the video domain, with significantly more computational cost and resources, Xu er al. [79] trained video variant of CLIP but excelled on a smaller number of downstream video tasks. This is partly attributed to two orders of magnitude more minor training data and limited availability of computing resources, large video data is notoriously more difficult to collect, manage, and process than image data. Under these restrictions, large pre- rreitrained image models are arguably still favorable in the selection of model initialization for video tasks.
In this work, we investigate a novel, critical problem of efficiently adapting large pre-trained image models for video downstream tasks, with a focus on the widely influential action recognition task. Considering that training video models is drastically more expensive in both computing resource and time than image models [19), this problem becomes particularly more useful and valuable in practice. On the other hand, it is also more challenging and non-trivial due to the extra necessity of overcoming the big gap between image and video transfer learning. Especially, pre-trained image models lack the ability to infer temporal structured information, which however is critical in video understanding. In fact, the key design with state-of-the art video models [00 6 is usually about learning the temporal dimension based on contemporary image models. Although model initializatton is still important, they largely go beyond the fine-tuning strategy, as architectural modification is often imposed in addition to full model training/fine-tuiing per downstream task.
Given that this a new problem, we first conduct a comprehensive benchmark using both various fine-tuning methods for image-to-video transfer learning and ffte-offthhear video models [6] 9].
Regarding the pretrained image model, we select two Vision Transformer (ViT) [T6] models, with one from CLIP pre- tre-training (55] and the other pre- trained on ImageNet-21K [14 VIT is representative in terms of network architecture, pre-training algorithm, and training data scale. Crucially, we further propose an efficient yet effective Space-Time Adapter (ST- (ST-Adapter), capable of extracting and leveraging the pre-trained knowledge large image model to achieve superior video understanding at small parameter cost. Specifically, ST-Adapter is formulated based on novel parameter-efficient bottleneck with sequence of operations including feature dimension reduction, spatial-temporal modeling, and feature dimension recovery. It is casy to implement and scalable for deployment since

---

all the primitive steps are realized with standard operators (e.g., fully-connected layer, depth-wise 3D convolution). With such lightweight design, our bottleneck can be cheaply integrated throughout the base network for enabling stronger layer-wise spatio-temporal learning. As a result, our model can be more rapidly optimized using fewer training epochs for significant convergence advantage.
We summarize the contributions as follows. (1) We investigate a new problem of parameter- efficient image-to-video transfer learning. Our motivation is to advocate the usability and deployment of increasingly larger whilst ever more powerful pre-trained image models in benefiting more challenging video understanding tasks. (2) We establish a benchmark for action recognition tasks by comprehensively experimenting with variety of fine-tuning strategies and several state-of -ot--o---herrr video understanding models. (3) We introduce novel gurameter-efficieee
Adapter (ST- SST-Adapter) for more effectively capitalizing large pre- pre-trained image model in video understanding. By grounding all the primitives on standard operators, ST-AAdatoor is easy to implement and friendly to deployment. (4) Extensive experiments on action recognition datasets show that our ST--Adppter outperforms not only existing parameter-efficien alternatives and the full fine-tuning strategy, but also state-of-the-art video methods with the same network architecture and model initialization.
2 Related Work
Parameter-efffcient transfer learning Driven by the wider application of large pre-trained language models across diversity of downstream tasks, the topic of efficient tuning has received increasing attention in NLP. Existing efficient tuning methods fall broadly into three categories. The first category is introduce task-specific adapters (25] 24 51 [50]. Specifically, an adapter consists of lightweight modules inserted between layers of a pre-trained model. To be parameter-effiieen, only those newly added adapter modules need to be updated during task fine-tuning, whilst all the parameters of the large pre-trained model, which takes the majority proportion of the whole solution, are frozen. The second category prompt tuning (39 52 31 62 42]. Instead of manipulating the network architecture, these methods prepend a set of learnable tokens at the input point of the model or intermediate layers. Similarly, only these added tokens need be optimized for each downstream task. The third category is learning weight approximation [27]. In particular, only the low-rank matrices for approximating the weights need to be updated during training.
Early works for efficient transfer learning in vision focus on parameter sharing in the context of multitask learning 83 57 (66.. Recently, there are several works for extending the efficient tuning idea from NI.P to vision tasks. CoOp 85 and CoCoOp [66] apply prefix tuning for adapting the CLIP model to various image recognition tasks. VI.-Adapter (65) achieves the performance comparable to full fine-tuning on challenging vision-language tasks. Commonly, their design focuses are all restricted to the text encoder of the CLIP model. More recently, (29 84) introduce the idea of prompt learning to visual backbones. They obtained favorable results on various image recognition benchmarks. Moving a step further, in this work, we consider the more challenging adaptation problem from pre- pre-trained image model without temporal knowledge to video understanding tasks.
Video action recognition Action recognition in the unconstrained video has largely been dominated by deep learning methods, thanks the availability of large video datasets, egg., Kinetics [10 and Something Swweethinn-Smmthingg (22). As a key component, the model architectures adopted by existing video methods has expanded from CNNS 32
68
19
T8
69
72
45 Transformers
17 40 38,
[6] As temporal information is important for modeling the dynamics, a variety
49 Further, different training methods have also been explored, e.g., unsupervised learning 667 20 76,, and vidco-text contrastive learning (64 79 78 66]. New opportunities for stronger video models are created following the introduction of large pretrained foundation models [55] 28 81. For example, Wang et al. 74 equipped the CLIP with temporal modules and good performance can be achieved after the model fully fine-tuned on video datasets. Ju et al. [3I] adopted the CLIP model for video recognition tasks by learning video- specific prompts. In contrast, in this work, we explore the potential of the large pre-trained image models with the parfmeter-efficient adapter strategy. Importantly, despite the simplicity, we bring about more significant advantages in performance along with new benchmark on pfrfmeter-efficient image-to- tinge-to-viieo video transfer learning.
of motion learning techniques has been introduced 75 30
3

---

3 Methodology
To capitalize a large pre-trained image model for more challenging video understanding such as action recognition in cross-mdaaitty manner, it is necessary to fill the intrinsic gap between image and video. For easier understanding, we start with an intuitive baseline based on temporal aggregation.
Temporal aggregation A straightforward baseline method of exploiting pre-trained image model for video understanding temporally aggregate per-frame feature representations (e.g.. average pooling). Concretely, given an input video clip V x where T, H, W are the number of frames, height and width respectively. Following [6), we first split each frame into N = H x W/ p2 patches of size P x P. Then, we flatten these patches and project them into a sequence of patch tokens Z. = [1,, [l....e........ [N 2, E Rdd where = 3 x p2 with t = I,... T. The sequence of feature vectors is then enhanced with the positional embedding by element-wise addition, along with a trainable class token concatenated. Subsequently, we feed each sequence with + tokens to a stack of self-attention based blocks individually. For each sequence keep only the classification token z,t² We further perform temporal average pooling on the class tokens final = $ Σ z4
to yield compact representation for the whole clip. We obtain the prediction by passing final through a classifier. As the sptial information is only naively averaged over time, it is also known as Space-Only TimeSformer [
Spatio-temporal attention For more dedicated structural modeling in the time dimension with ViTs, a mainstream approach in the video domain is to develop various spatio-temporal attention mechanisms by further imposing temporal attention on top [6 2 9 82 23 We choose two representative video ViT models, TimeSformer [6] and XVTT 19. in our performance benchmark. However, state-of-the-an video ViT models often need to fully fine-runed per task, which is parameter- inefficient, given that in this way we have to keep a separate copy of the whole fine-tuned model parameters for every single task.
3.1 Preliminaries
Our method is inspired by the Adapter (25] designed for parameter-efficient transfer learning in NLP. Specifically, the adapter module is composed of down-projection linear layer followed by a non-linear activation function and an up-projection linear layer. Formally, given an input feature matrix X RNxd at the i-th layer, the feature adaptation process can be written
Adapter (X)= X + f(X W down
upt
where W Woown E Rxx refers to the down projection layer, W up E R the up-projection layer, and f()) the activation function. Note, that residual summation applied for preserving the information in input as required. The idea of Adapter has been remarkably successful in NLP due to several advantages: (1) High parameter efficiency across tasks since only a small number parameters are task-specific; (2) Reaching on-par performance compared to full fine-tuning; (3) Taking significantly small training costs; (4) Avoiding the catastrophic forgetting limitation of full fine-tuning.
We aim to propagate the success of Adapter from NLP to computer vision particularly the image- to-video transfer learning problem as discussed earlier. To that end, we introduce novel Adapter tailored specially for spatio-temporal reasoning a key capability for video understanding which, however, existing NLP Adapter variants lack.
3.2
Adapter (ST-Adapter)
Typically, an image model only considers the ability of spatial modeling. The objective of our Adapter (ST-Adapter) is to enable pre-trained image model to reason about spatial and temporal information of video in a parameter efficient principle. design, we consider a couple of practically-crucia criteria: (1) Smaller parameter size: The parameter cost for each downstream task should be small the essential criterion for parameter efficiency. (2) Development friendllines:: This is critical for real-world development and deployment. In practice, it is necessary that a model can be easily implemented using the standard highly optimized deep learning toolboxes (e.g., PyToch, TensorFlow, TensorRT, and TorchScript), without tedious per-toolbox specialization.

---

This also facilitates the realization of high inference efficiency across a diversity of running platforms due to the best usage built-in software and hardware resources.
Under these considerations, we formulate the proposed ST-Adapterr by sticking to commonly-adoppeed primitive operators alone. Starting with the above Adapter (Eq. (T)) originally developed for NLP tasks, we further introduce a spatio-temporal operator realized by a standard depth-wise 3D- convolution layer [8] between the bottlenecks (Figure []. particular, our spatio-temporal operator enables layer-wise temporal inference efficiently, because it only operates in a compressed low- dimensional (e.g.. 128D) feature space and the depth-wise convolution is highly efficient both in parameter and computation [26]. As a result, this yields an introduction of tiny extra (~~2%) parameters and (~0.3%) computation. Formally, our ST-Adapter can be expressed as:
ST ST-Adapter X) = X + DWConv3D (X down) W up1
(2)
where DWConv3D denotes the wist 3D-convolution for spatio-temporal reasoning we introduce. It is noteworthy that before applying DWConv3D, the down-projected feature representations will be first reshaped from X' E T xd to X" E Ix''xxxxxd (where N = h x w) to have the spatial and temporal dimensions prepared for reasoning. With this highly integrated design, our ST-Adapter enjoys the same efficiency and flexibility as the NLP Adapter, while uniquely being able to conduct spatio-temporal modeling. I.
3.3 ST-Adapter Integration
For proper adaptation, the adapter modules are often integrated between layers of Transformer. In NLP, a variety of integrating designs have been investigated. For example, [25] deploys two adapter modules per layer with following the Multi-Head Self-Attention (MHSA) and the other following the only one adapter after the FNN suffices. Similarly, our ST--Adapter
Networks (FFN) [25]. On the other hand, 663 5 suggest that adding
be also integrated generally distinctive positions. Empirically, we find that decent performance can be achieved in case a single ST- ST-AAapter is placed before the MHSA of each transformer block (Figure a[a and Table[cc).
Experiments
4.1 Experiments Setup
Datasets For the benchmark experiments, we use two popular video action recognition datasets.
Kinetics-400 (K400): The K400 [33] dataset contains 244k training videos and 20k validation videos labeled with 400 action categories. Most videos have a length of 10s or about 300 frames. While there is great diversity in these videos, they largely biased spatial appearance (60)
Something- Something-Somettinn Something-v2 (SSv2): The SSv2 [2 dataset consists of 220,487 videos covering 174 human actions. The video length ranges from 2 to 6 seconds. In contrast to K400, SSv2 presents richer temporal information with much higher significance [60].
Epic-K OpiicKKicchens-a 100 (EK100): The EK100 13 dataset consists of 100 hours of video in egocentric perspective recording a person interacting with variety of objects in the kitchen, Each video sample is labeled with a verb and a noun. We report top-1 verb and noun classification accuracy.
Pre-trained models In all experiments, we use the standard ViT [16] as our base backbone model. We conduct most of our experiments with the VIT-B/16 variant with layers and 86M parameters, taking input a sequence of patches at size 16 x 16.
What was learned during pre-training directly decides the knowledge that can be transferred to downstream tasks, thus also the effectiveness upper bound of transfer learning methods. To this end, we benchmark the same backbone under two different pre-training strategies: pre-training with web-scale raw data that has been recently proposed by CLIP [55] (400M image-text pair) and classical supervised pre-training on annotated data from ImageNet-2111 -1K (21k classes and 14M images).
Implementation details. All details, including training and testing settings and module instantiation details, provided in the appendix.
5

---

Competitors We provide several transfer learning approaches in our benchmark for efficient image- to-video transfer learning. Note that the parameters of the linear classifier are always updated during training for all approaches.
(1) Full Fine-tuning: Fully updating all the parameters when adapting for a specific target task,
(2) Partial Fine-tuning: Only update the last ViT layer while keeping the rest of the parameter fixed.
(3) Temporal Fine-tuning: We only tune the temporal attention modules (i.e., TA) in the SA+TA architecture.
(4) Linear Probing:
Freezing all the parameters except those in the linear classification layer.
Adapter 25
Adding small sub-networks between layers of pre-trained model. During fine-tuning, we only update the newly added parameters introduced by the adapters.
(6) Prompt Tuning [9]]: Prepending a sequence of learnable prompt tokens to the input visual patch tokens. During fine-tuning, only these newly added prompts are updated.
(7) Attention Pooling Head: Replacing the original temporal average pooling with a temporal attention pooling layer (similar to the one used in [9] before the classification head.
These approaches above do not incorporate temporal modeling to the image ViT. Hence, we further consider temporally augmented ViT architectures as introduced in state-of-the-art video methods:
(a) Spatial Attention Only (SA): Space-Only TimeSformer
6
(b) Spatial Attention + Temporal Attention (SA+TA): The default TimeSformer 6 with divided space-time attention (Fig. (a).
(c) Spatial Attention Temporal Shift (SA+TS): XVIT 9
Note that not all fine-tuning protocols are compatible with each of these video ViT variants. Take SA+TS for example, the original model behavior is altered with channel shift, as a result, is not compatible with Linear Probing that requires freezing all the parameters of the backbone.
4.2 Main Results and Analysis
Cross-modality fine-tuning benchmark. Table I presents the results of fine-tuning ViT-B/16 pre-trained with CLIP and ImageNet-2IK. All baselines are built by combining existing efficient fine-tuning methods with three state-of-hhe-arl ViT-based action recognition models. From the results we can see that:
(i) For CLIP pre-trained model, ST-Adapter performs par with Full Fine-tuning (82.0 vs. 81.7 for K400 and 66.3 vs. 66.1 for SSv2) while updating far less parameters (7.2M vs. 121.57M). ST-Adapter significantly outperforms all other efficient fine-tuning methods. We see that baselines like Prompt Tuning and Partial Fine-tuning can provide non-trivial gain performance compared to Linear Probe, but are still behind our ST-Adapter.
(ii) Our ST-Adapter can generalize across different pre-training datasets and methods. We can see that CLIP pre-train models dominate over ImageNet-21K K pre-train ones. These results well match the shift of paradigm in current AI research 77), where pre-training no longer needs limiting to curated data and annotations to deliver good performance on downstream tasks, but can take advantage of broader scale web raw data.
Interestingly, we observe that SSv2, a motion-centric dataset in design, also benefits from stronger appearance (image) pre-training. We think this may attribute to that raw textual description can provide a much richer description (i.e., human-object relations) of the image than curated limited categorical labels. Full fine-tuning on SA+TS (XVIT) performs slightly worse with CLIP pretrain than ImageNet-21k 1k pretrain. We conjecture this is because the channel shift operation breaks the knowledge in the pre-training weights, and thus does not benefit much from stronger pre-training like CLIP.
Comparison to the state-of-the-art models. We compare ViT with ST-Adapter to other state-of-th of- arts methods on both K400 dataset [33] in Table 2 SSv2 dataset [22 in Table Bl and 10 dataset 13 in Table We can observe that:
(i) With the proper adaptation method, we can simply turn large image foundation model into good video model by only tuning few parameters, Our results comparable to or better than previous

---

Table 1: Benchmark results on Kinetics-400 and Something-Something-22 We evaluate all the approaches on two datasets with VIT-B/16 pretrained with CLIP and ImageNet-213 21K. For each entry, we report the topl action recognition aceuracy and the number of fine-tuned parameters. methods introduce extra parameters beside parameters of the VIT backbone and linear classifier. Our ST-Adapter achieves the best trade-off between accuracy and training efficiency. It is the only efficient fine-tuning method that can match the performance of full fine-tuning. The TM? column shows whether the method includes temporal modelling, i.e., a temporal aggregation method other than average pooling. All models are trained using 8 frames and tested with views.
CLIP
ImageNet-2IK
Fine-tuning Methods
Fine-tuned Params (M)
Architecture
TM?
K400
SSv2
K400
SSv2
SA
86.1 1
81.0
44.0
76.9
40.0
Full Fine-tuning
SA + TA 6
121.57
81.7
66.1
78.0
59.5
SA + TS
93.79
78.0
62.0
78.5
64.4
SA
7.40
80.1
37.6
61.7
20.4
Partial Fine-tuning
SA TA
10.36
80.3
57.5
63.1
29.3
Temporal Fine-tuning
SA + TA
35.8
81.3
59.4
76.5
51.9
Prompt Tuning
SA
1.18
79.3
39.3
71,4
26.3
Attentional Pooling
SA
2.36
75.3
21.5
59.1
15.1
Linear Probe
SA
0.31
76.6
21.9
60.1
14.8
Adapter 25
SA
6.77
81.6
46.2
76.2
40.5
ST-Adapter (ours)
SA
7.20
82.0
66.3
76.6
62.8
methods tailored for such tasks. Our largest model with ViT-L backbone set new state-of-the-art in K400 by achieving 86.7% top-1 accuracy.
(ii) is noteworthy that, our method takes significantly fewer frames as input compared to other methods (8 vs. 16. 32, 64. 96). It is also reflected in terms of GFlops. Saying that the VIT was not designed for efficiency purposes like [38 accuracy-efficiency trade-offs.
but the adapted CLIP ViT has achieved similar
(iii) The paradigm of pre-training and fine-tuning has been widely adopted in most state-of-art methods to achieve good performance. Between them, most of the approaches start from image pre-trained models, and only a few can afford video pre-training. Note that for the Something- Something dataset, except MVIT [17 pre-trained video data from scratch, the rest of methods are still initialized from image pre-trained weights. A good image pre-trained model with rich appearance information can facilitate temporal modeling in temporally challenging datasets like SSv2.
(iii) It evident in Table 4 that our ST-Adapter consistently brings a big margin on egocentric videos. Also, we found that without our ST--Aaater, it is much more difficult to directly adapt CLIP pre-trained ViT on the domain of egocentric video with high sensitivity to the hyper- parammer setting. ST-AAapteer eases the training process. It is worthy to note that, all current transformer based approaches need to be pre-trained first on image dataset and then fine-tuned on Kinetics dataset before fine-tuned with egocentric videos. In contrast, our ST-Adapter can be directly applied to an image model and trained with target egocentric video alone.
4.3 Ablations
Unless otherwise specified, use ViT-B/16 backbone and input frames in all ablation experiments, and we use one ST-Adapter with bottleneck width 384 before MHSA in each Transformer block.
Where to insert ST-Adapter By default, we insert a ST-Adapter to every Transformer block in the backbone, but we also show the performance impact of using fewer ST-Adapters. As shown Table 5b while more ST-Adapters tend to do better, ST-Adapters deeper layers boost performance more
7

---

Table Results on Kinetics-400 validation set. "Frames" denotes the total number of frames used during inference which is: # frames per clip # temporal clip x # spatial crop. "GFlops" means 10 Flops. Our ViT ST-Adapter achieves new state-of-the-are performances on K400 at similar GFlops.
Model
Pretrain
#Frames
GFlops
Top-1
Top-5
Methods with full full-finetuning
LGD
IN-IK
128xN/A
N/A
79.4
94.4
SlowFast+NNL. 101++LL11
16x3x 10
7020
79.8
93.9
ip-CSSN 70
Sports 1M
32x3x10
3270
79.2
93.8
CorrNet
Sports
32x3x 10
6720
81.0
X3D-XI. 18
16x3x10 10
1452
79.1
93.9
MoViNet-A6
120x1x1
386
81.5
95.3
ViT-B-VTN 47
IN21K
250x1x1
3992
78.6
93.7
TimeSformer-L(0
IN21K
96x3x1
7140
80.7
94.7
STAM 61
IN21K
64x1xl
1040
79.2
X-VIT 9
IN2IK
16x3x1
850
80.2
94.7
Mformer-HR 49
IN-21K
16x3x100
28764
81.1
95.2
MVVT-B,32x311
32x1 x
850
80.2
94.4
ViViT-L/22
JFT300M
16x3x4
17352
82,8
95.3
Swin-B 44
INIK
32x3x4
3384
80.6
94.6
Swin-L(384) 44
IN21K
32x5x10
105350
84.9
96.7
UniFormer-BB
INIK
32x1x4
1036
82.9
95.4
VATT-Large(320)1
How HloTTo10M
32x3x4
29800
82.1
95.5
TokenL_earner(55
JFT300M
64x3x4
48912
85.4
96.3
OMNIVORF((Win-L))2
IN22K+SUN
32x3x4
7248
84.1
96.3
MTV
WTS-
32x3x4
73570
89.9
98.3
VIT-B w/o ST-Adapter
CLIP
8x3x1
419
81.0
95.5
ViT-L, w/o ST-Adapter
CLIP
8x3x1
1941
85.8
97.2
Methods with frozen backbone
Our ViT-B w/ ST-Adapter
CLIP
8x3x1
455
82.0
95.7
Our ViT-B w/ ST-Adapter
CLIP
16x3x1
911
82.5
96.0
Our ViT-B w/ ST-Adapter
CLIP
32x3x1
1821
82.7
96.2
Our VIT-L w/ ST--Aapteer
CLIP
8x3x1
2062
86.7
97.5
VIT-L ST-Adapter
CLIP
16x3x1
4124
86.9
97.6
Our VIT-L w/ ST-Adapter
CLIP
32x3x1
8248
87.2
97.6
than those at shallower layers. This observation is useful when we insert ST--Aappters into deeper models and having Adapter for each block might be expensive. We also show the performance when inserting ST-Adapters to different positions within a block. As shown Table 5c while the performance relatively insensitive the position of the Adapters, using multiple adapters one block may substantially boost performance on some datasets, like SSv2 in our case.
Training parameter efficiency We experiment with a different number of channels in the middle of the bottleneck design. As shown Table 5a and Fig. 2a our method is effective with wide range of bottleneck width: even with a channel reduction to 64, our ST--dapterrs still obtain relatively good performance, outperforming all baselines in Table exxeet for Full Fine-tuning (SA TA). Even with a bottleneck width of 768, our ST-Adapters are still very parameter efficient, introducing only about Transformer encoder block. In contrast to the inverted bottleneck design commonly used with depthwise convolutions [59], ST-Adapters work best with regular bottlenecks. The success of transfer learning with such low-rank projections again shows the rich knowledge and strong potential of modern foundation models.
1/6 new parameters
Training time efficiency In Fig. 2b we show an enlarged difference between full fine-tuned models and our ST-Adapters with low training budgets. When we reduce the number of training steps, the accuracy of full fine-tuned models drops significantly faster than models with ST-Adapters. This shows the advantage of our proposed modules when backbone models are large or computational resources are limited. We also report the total training GPU-hours and peak memory usage for three models: TimeSformer, VIT-B/16, ViT-B/16 with ST-Adapter (8 input frames, 16 samples per GPU on 8 V100 GPUs) in Table 6

---

Table Results on Something-Somethinn-222 validation set. "Frames" denotes the total number of frames used during inference which is: # frames per clip x # temporal clip x # spatial crop. "GFlops" means 10 Flops. Our ViT w/ ST-Adapter outperforms most of the current methods by only fine-tuning very small set of parameters. Here the VTT-B w/ ST-Adapter result is reported using ST-Adapters per block.
Model
Pretrain
#Frames
GFlops
Top-1
Top-5
Methods with full full-finttuning
TSM
INIK
16x1x1
66
63.3
88.5
GST
INIK
16x1x1
59
62.6
87.9
MSNet 35
INIK
16x1 x
101
64.7
89.4
CT-Net 37
INIK
16x1x1
75
64.5
89.3
TDN
INIK
16x1x1
72
65.3
89.5
TimeSformer-HR/G
IN2IK
16x3x1
5109
62.5
X-ViT(9
IN21K
32x3x1
1270
65.4
90.7
Mformer-I
IN21K+K400
32x3x1
3555
ViVIT-L 2
IN21K+K400
16x3x4
11892
65.4
89.8
MVIT-B-24.32x3111
K600
32x1x3
708
68.7
91.5
Swin-B 4
IN21K+K400
32x3x1
963
69.6
92.7
UniFormer-Bi
IN1K+K600
32x3x1
777
71.2
92.8
OMNIVORE (Swin- (Swin-B)(21 B)
IN22K*K40++SUh
32x3x1
963
71.4
93.5
MTV-B(320p)B8
IN2IK+K400
32x3x4
11160
68.5
90.4
VIT-B w/o ST-Adapter
CLIP
8x3x1
419
44.0
77.0
VIT-L w/o ST-Adapter
CLIP
8x3x1
1941
48.7
77.5
Methods with frozen backbone
Our ViT-B ST-Adapter
CLIP
8x3x1
489
67.1
91.2
Our VIT-B w/ ST-Adapter
CLII
16x3x1
977
69.3
92.3
Our ViT-B ST-Adapter
CLIP
32x3x1
1955
69.5
92.6
Our VIT-L w/ ST-Adapter
CLIP
8x3x1
2062
70.0
92.3
Our ViT-L w/ ST-Adapter
CLIP
16x3x1
4124
71.9
93.4
Our ViT-L ST-Adapterr
CLIP
32x3x1
8248
72.3
93.9
Table 4: Results on Epic-Kitchens-100 validation set. "Frames" denotes the total number of frames used during inference which is: frames per clip temporal clip # spatial crop.
Model
Pre-train data
#Frames
Verb
Noun
Methods with full-finetuning
ViVIT-I. 2
IN21K+K400
16
x
3 X 10
66.4
56.8
MFormer-B 49
IN21K+K400
16
3
X 10
66.7
56.5
XViT(8x) 9
IN21K+K400
8 x
3
X 1
66.7
53.3
ViT-B/16 w/o ST-Adapter
CLIP
X 3
54.8
50.4
Methods with frozen backbone Our ViT-B/16 w/ ST-Adapter
CLIP
x 3 x
67.6
55.0
Table Ablation study on K-400 and SSv2. (a) We show the performance with different channel
numbers in the bottleneck. (b) We evenly divide the 12 blocks of VIT-B/16 into 3 groups. Block no. I
closest to input and no. 12 closest to output. (c) Effect of where to put the ST-Adapter inside
block, whose diagram is shown in Fig.
(a) Bottleneck width
(b) Global position
(c) Local position
width
K400
SSv2
1-4
5-8
9-12
K400
SSv2
position
K400
SSv2
64
81.4
64.4
77.7
45.9
before MHSA
82.0
65.6
128
81.6
64.9
80.0
60.9
after MHSA
81.9
65.7
256
81.8
65.5
81.3
62.8
after FFN
81.9
65,9
384
82.0
65.6
81.8
65.6
before after
82.0
67.0
768
81.9
65.5
/
82.0
65.6
MHSA

---

61.7
Kinete --t0 atcurry
861
Roveria a00 scarsey P1
81
50.9
Kinetcss 0p0tcc--000000 400 acoraay
76.9
94.1
7/
79-
$0.1
70
69.4
78
Full mreeiuee
SlAAcaper (ours)
88
Partiel Plre-tune
Rooling
Probe
ST-Acapter tours]
58.1
ST-Adapter (nurs)
75
15000
treining
pettentinge
(a) Parameter Efficiency
(b) Training Efficiency
(c) Data Efficiency
Figure 2: Ablation study on efficiency (a) Parameter efficiency: ST-Adapter (with different bot-
tleneck width) is compared with efficient fine-tuning methods in Table [ (b) Training efficiency: We compare ST-Adapter with Full fine-tuning under different training schedules. Batch size aligned and their original schedules are shortened proportionally. (c) Data efficiency: Performance comparison on different training data scales. The same ViT-B/16 with CLIP pre-training is used for all experiments.
Training data efficiency Fig. 2c showcases the impact of training data size on action recognition accuracy. Even with the same pre-trained weights, ST--Adaperr tend to obtain higher performance than full fine-tuning especially on smaller datasets: the margin between the two models increases with the shrinkage of data. This shows that ST-Adapters are powerful tools to transfer to downstream tasks where only small amount of labeled data is available.
Effects of kernel shape proposed ST-Adapter, It is shown in Table 7 that the temporal span is most sensitive, suggesting the significance of temporal structural modeling we focus on in this work.
We ablate the effect of kernel size in the depth-wise convolutions inside our
Table 6: Training time and memory. For full-fneetuning we used the recipes in 66
Model
Training GPU-hours (K400)
Peak mem (MB)
TimeSformer 6 (Full Fine-tune)
60 (+161%)
21694 (+52%)
ViT-B/16 (Full Fine-tune)
40 (+74%)
17275 (+21%)
ViT-B/16 w/ ST-Adapter
23
14238
Conclusions
In this work, have presented simple yet effective Spatio- Temporal Adapter (ST-Adapter) for enabling a less studied parameter-efficient image-to-video transfer learning. Fully us- ing commonly adopt primitive operators, ST-Adapter is particu- larly designed to be both lightweight and easy implement for friendly usability and deployment. This cross mrods-mo adap- tation is practically critical capability considering that is dramatically challenging and more costly to build a sufficiently strong large video model in reality. Encouragingly, extensive experiments on video action recognition show that our ST-
Table 7: Effects of kernel shape.
Kernel size denoted as ky ky x kw for time, height and width.
Kernel Size
K400
SSv2
1 1 x
81.6
46.2
1x3x3
81.4
46.2
3 3 x
82.0
66.3
x
82.0
65.6
Adapter can match or surpass both the full fine-tuning strategy as well as fully trained state-of-the-art video models, whilst having the benefit of (20 times less updated parameters) parameter-efffiiencyy Further, our method is also faster to train and consumes less computing resources with economic and environmental superiority. We believe this work is inspiring for the research of other video understanding tasks such as action localization and video summarization.
Acknowledgement This work is supported in part by Centre for Perceptual and Interactive Intelli- gence Limited, in part by the General Research Fund through the Research Grants Council of Hong Kong under Grants (Nos. 14204021, 14207319).
10

---

References
[1]
Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Roqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Newral Information Processing Systems, 34, 2021.
[2] Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučié, and Cordelia Schmid. Vivit; A video vision transformer. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 6836-6846, 2021.
[3] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 1607.0645000 2016.
[4] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Visual prompting: Modifying pixel space to adapt pre-trained models. arXiv preprint arXiv:2203 2022.
[5] Ankur Bapna, Naveen Arivazhagan, and Orhan Firat. Simple, scalable adaptation for neural machine translation. arXiv preprint 19Xxvv:999.0080 2019.
[6] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. space-time attention all you need for vidco understanding? arXiv preprint arXiv:2102.0500 2021.
[7] Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
[8] Tom Brown, Benjamin Mann. Nick Ryder. Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss,, Gretchen Krueger, Tom Henighan. Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bernr,, Sam McCandlish, Alec Radford, lyya Sutskever, and Dario Amodei. Language models are fow-shot learners. volume 33, pages 1877-1900 1901. Curran Associates, Inc., 2020.
[9] Adrian Bulat, Juan Manuel Perez Rua, Swathikiran Sudhakaran, Brais Martinez, and Georgios Tzimiropou- los. Space-time mixing attention for video transformer. Advances in Neural Information Processing Systems, 34, 2021.
[10] Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299-6308. 2017.
[11] Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600.. arXiv preprint arXiv: 00.440 2018.
[12] Joao Carreira, Eric Noland, Chloc Hillier, and Andrew Zisserman. A short note on the kinetics-708 700 human action dataset. arXiv preprint 1rXiv:1907.06990 2019.
[13] Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Antonino Furnari, Jian Ma, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro. Toby Perrett, Will Price, and Michael Wray. Rescaling egocentric vision: Collection, pipeline and challenges for epic-kitchens 100. International Journal of Computer
Vision (IJC),, 130:33-55 55, 2022. URL (tLhstt:::: / //oii org/ 10. 1007/s111100 021-1226---22-111111
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009.
[15] Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
[16] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Hcigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. 2020.
[17] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. arXiv preprint 1rX1v22104.1122, 2021.
11

---

[18] Christoph Feichtenhofer. X3d: Expanding architectures for efficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 203-213, 2020.
[19] Christoph Feichtenhafer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1202-6211, 2019.
[20] Christoph Feichtenhofer, Haoqi Fan, Bo Xiong, Ross Girshick, and Kaiming He. A large-scale study on unsupervised spatiotemporal representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3299-3309, 2021.
[21] Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16102- 16112, 2022.
[22] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Hcuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision, pages 5842 5850, 2017.
[23] Ryota Hashiguchi and Toru Tamaki. Vision transformer with cross-attention by temporal shift for efficient action recognition. arXiv preprint arXiv: 00452, 2022.
[24] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor view of parameter-efficiee transfer learning. 2022.
and Graham Neubig. Towards a unified
[25] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In ICML, pages 2799-2799, PMLR, 2019.
[26] Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv:17044 04861, 2017.
[27] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, arXiv preprint arXiv:2106. 09685, 2021.
[28] Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun- Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and rision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4911 PMLR, 2021.
[29] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Scr-Nam Lim. Visual prompt tuning. arXiv preprint arXiv:2203.1 12119, 2022.
[30] Boyuan Jiang. MengMeng Wang. Weihao Gan, Wei Wu, and Junjie Yan. Stm: Spatiotemporal and motion encoding for action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2000-2009, 2019.
[31] Chen Ju. Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual- lanuull models for efficient video understanding. arXiv preprint arXiv:2112.04478, 2021.
[32] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei. Lcage-scale vidco classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1725 1732, 2014.
[33] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vjjayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action vidco datasct. arXiv preprint arXiv:/705.000 06950, 2017.
[34] D. Kondratyuk, Liangzhe Yuan. Yandong Li. Li Zhang, Mingxing Tan, Matthew A. Brown, and Boqing Gong. Movinets: Mobile video networks for efficient video recognition. ArXiv, 111/2103.11111.. 2021.
[35] Heeseung Kwon, Manjin Kim, Suha Kwak, and Minsu Cho. Motionsqueeze: Neural motion feature learning for video understanding, In ECCV. 2020.
[36] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
12

---

[37] Kunchang Li. Xianhang Li, Yali Wang. Jun Wang, and Y. Qiao. Ct-net: Channel tensorization network for video classification. ArXiv, abs/2106.01603. 2021.
[38] Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song. Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. arXiv preprint arXiv:2201. 009450, 2022.
[39] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pages 4582- 4597, Online, August 2021. Association for Computational Linguistics.
[40] Yanghao Li, Chao-Yuan Wu, Haoqi Fan, Kartrikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Improved multiscale vision transformers for classification and detection. arXiv preprint urXiv:2112.01526. 2021.
[41] Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7083-7093, 2019.
[42] Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du. Zhilin Yang, and Jie Tang. P-tuning v2: Prompl tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint 4rXiv:2110.07602 2021.
[43] 7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. 2021.
[44] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. arXiv preprint arXiv:2106. 13230, 2021.
[45] Zhaoyang Liu, Limin Wang, Wayne Wu, Chen Qian, and Tong Lu. Tam: Temporal adaptive module for video recognition. arXiv preprint arxiv22005. 06803, 2020.
[46] Chenxu Luo and Alan L. Yuille. Grouped spatial-temporal aggregation for efficient action recognition.
2019 IEEE International Conference on Computer Vision (ICCV), pages 5511-5520, 2019.
[47] Daniel Neimark, Omri Bar, Maya Zohar, and Dotan Asselmann. Video transformer network. ArXiv, abs/2102.00711 2021.
[48] Junting Pan, Siyu Chen, Mike Zheng Shou, Liu, Jing Shao, and Hongsheng Li. Actor-context-actor relation network for spatio-temporal temporal action localization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 464 474, June 2021.
[49] Mandela Patrick, Dylan Camphell, Yuki Asano, Ishan Misra, Florian Metze, Christoph Feichtenhofer, Andrea Vedaldi, and João F Henriques. Keeping your eye the ball: Trajectory attention in vidco transformers. Advances in Neural Information Processing Systems, 34, 2021.
[50] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: dentdusttuetive task composition for transfer learning. arXiv preprint arXiv:2005 002477, 2020.
[51] Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulié, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP 2020): Systems Demonstrations, pages 46-54, Online, 2020. Association for Computational Linguistics.
[52] Guanghui Qin and Jason Fisner. Learning how to ask: Querying Ims with mixtures of soft prompts. arXiv preprint arXiv: 06599, 2021.
[53] Zhaofan Qiu, Ting Yao, C. Ngo, Xinmei Tian, and Tao Mei. Learning spatio-temporal representation with local and global diffusion. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12048 12057, 2019.
[54] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language models are unsupervised multitask learners. OpenAl blog, 188),, 2019.
[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 8748-8763, PMLR,
2021.
13

---

[56] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. 30, 2017.
[57] Sylvestre- Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Efficient parametrization of multi domain deep neural networks. pages 8119 8127, 2018.
[58] Michael Ryoo, AJ Piergiovanni, Anurag Arna,, Mostafa Dehghani, and Anelia Angelova. Tokenlcamnr: Adaptive space-time tokenization for videos. Advances in Neural Information Processing Systems, 34, 2021.
[59] Mark Sandler. Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chich Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4510-4520, 2018.
[60] Laura Sevilla-Lara, Shengxin Zha, Zhicheng Yan, Vedanuj Goswami, Matt Feiszli, and Lorenzo Torresani. Only time tell: Discovering temporal data for temporal modeling. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 535-544, 2021.
[61] Gilad Sharir, Asaf Noy, and Lihi Zelnik-Manot An image worth 16x16 words, what a video worth? ArXiv, abs/2103.139) 13915, 2021.
[62] Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts, arXiv preprint arXiv:2010. 15980, 2020.
[63] Asa Cooper Stickland and lain Murray. Bert and pals: Projected attention layers for efficient adaptation in multi-task learning. In International Conference on Machine Learning, pages 5986-5995. PMLR, 2019.
[64] Chen Sun, Austin Myers, Carl Vondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7464-7473, 2019.
[65] Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VI-adapter: Parameter-efficient transfer learning for vision- and-language tasks. arXiv preprint urXiv:2112.0683 2021.
[66] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In European conference on computer vision, pages 776-794. Springer, 2020.
[67] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training. arXiv preprint arXiv:2203. /2602, 2022.
[68] Du Tran. Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference computer vision, pages 4489-4499 2015.
[69] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look spatiotemporal convolutions for action recognition. Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450-6459, 2018
[70] Du Tran, Heng Wang, L. Torresani, and Matt Feiszli. Video classification with channel-separated convolu- tional networks. 2019 IEEE/CVF International Conference Computer Vision (ICCV), pages 5551-5560, 2019.
[71] Heng Wang, Du Tran, L. Torresani, and Matt Feiszli. Video modeling with correlation networks. 2020 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 349-358, 2020.
[72] Limin Wang. Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Lue Van Gool. Temporal segment networks for action recognition in videos. IEEE transactions on puttern analysis and machine intelligence, 41(11))2740-277 2704000--7552200 2018.
[73] Limin Wang, Zhan Tong, Bin Ji, and Gangshan Wu. Tdn: Temporal difference networks for efficient action recognition. ArXiv, a. 10077, 2020.
[74] Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08477 2021.
[75] Xiaolong Wang, Ross Girshick. Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 2018.
14

---

[76] Chen Wei, Haoqi Fan. Saining Xie, Chao-Yual Wu, Alan Yuille. and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. arXiv preprint arXiv:2112.09133, 2021.
[77] Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, and Kevin Murphy. Rethinking spatiotemporal feature learning: Speed-accuracy trade-off's in video classification. In Proceedings of the European conference on computer vision (ECCV), pages 305-321, 2018
[78] Hu Xu, Gargi Ghosh, Po- Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, and Luke Zettlemoyer. Vlm: Task-agnostic sideo-language model pre- pre-training video understanding. arXiv preprint 11Xv:2105.00990 2021.
[79] Xu, Gargi Ghosh, Yao Huang. Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. Videoclip: Contrastive pre-training for zero-shot video-text understanding. In EMNLP, 2021.
[80] Shen Yan, Xuehan Xiong. Anurag Arnab, Zhichao Lu, Mi Zhang. Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 3333-3343, 2022.
[81] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint wwXiv:2205.01117 2022.
[82] Hao Zhang, Yanbin Hao, and Chong-Wan Wah Ngo. Token shift transformer for video classification. In Proceedings of the 29th ACM International Conference Multimedia, pages 917-925, 2021.
[83] Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side-tuning: baseline for network adaptation via additive side networks. pages 698 714. Springer, 2020.
[84] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. Neural prompt search. arXiv preprint arXiv:2206.04673, 2022.
[85] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. arXiv preprint 13Xi::2109.0113, 2021.
[86] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In CVPR, 2022.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs' of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Code will be provided on GitHub after blind review.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See ??
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [No] The experiments are too expensive to repeat many times.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] All are mentioned in
(b) Did you mention the license of the assets? [No]
(c) Did you include any assets either in the supplemental material URL? [Yes] Code will be provided on GitHub after blind review.
(d) Did you discuss whether and how consent was obtained from people whose data you'le 're using/curating? [No]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "xjkFZsa49R", "reviewer": "Reviewer_Y5Pa", "summary": "The goal of this paper is to perform parameter-efficient learning on image-to-video adaptation tasks. Specifically, the authors propose a Spatio-Temporal Adapter, which applies an additional depth-wise 3D convolution in adapters, to incorporate temporal and spatial information with only a limited amount of parameters in the model fine-tuned. In the experiment, the authors showed that, compared with full fine-tuning and other fine-tuning techniques, the proposed ST-adapter can achieve better results in Kinetics-400 and Something-Something-v2 datasets.", "questions": "The following questions are mainly based on the concerns listed in the Weaknesses part. Here are the questions:\n\n- I have some concerns regarding the experimental setup (i.e., doing early-stoping on the validation set and only presenting validation set results for parameter-efficient methods). Could the authors provide the test set results (or doing early-stopping on the held-out set) or provide arguments to alleviate this concern?\n\n- Could the authors present a column to clearly show the number of total parameters across different methods? How does the extra parameter affect the inference time compared to the method without the extra parameters?\n\n- To have a fairer comparison to SoTA, could the authors provide the results under the same setups, such as results of (1) ViT-L pretrained on IM-21k and applying ST-Adapter and (2) ViT-L pretrained on Clip and without ST-adapter?\n\n\n- Notation is confusing. In line 120, the authors mention $N=H \\times W / P^2$ which is the number of patches in a single frame without temporal information. However, in line 173, the authors mention the feature representations will be first ***reshaped*** from $X' \\in R^{N \\times d}$ to $R^{T \\times H \\times W \\times d}$. Why does $N$ contain the temporal information? There seems to be notation confusion or inaccurate description. \n\n", "limitations": "I do not see the authors including any discussion on limitations or negative societal impact. The potential limitations of the paper are the inference time of using extra parameters and generalization to other video tasks. ", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The motivation is reasonable. Since the video pre-training data or pretrained models are limited or hard to obtain, it is ideal to use pretrained models/data from other modalities and adapt to video domains.\n\n- The paper is easy to understand, and the presentation is clear and good.\n\nWeaknesses:\n- Since the major difference between the prior work [1] and the proposed method is depth-wise 3D convolutional layers in the hidden state, it is important to compare the prior work and show that such a design makes a difference in the performance. \n\n- A major weakness of this paper is the experimental setup. As mentioned in the prior empirical study [2] on parameter-efficient methods, a common pitfall or mistake of the existing parameter-efficient works is only presenting validation set results while doing early-stoping on the validation set. This makes the reported results of the parameter-efficient methods biased on the validation set and higher than the full fine-tuning. A more reasonable experimental setup is to either report test set results or split the training set and perform early-stoping on the held-out set. It is unclear whether this paper follows the proper experimental setup. \n\n- In Table 1, only showing the number of fine-tuned parameters is a bit misleading, as some methods including ST-adapter introduces **extra** parameters which makes the comparison unfair due to the different amounts of total parameters in the models across different methods. \n\n- Comparison to the state-of-the-art methods is confusing. Since the major focus is the effectiveness of the ST-adapter, a fair comparison should be, under the same experiment setups (e.g., same pretrained datasets), how the ST-adapter improves against the prior works. For example, the authors should present the results of (1) ViT-L pretrained on IM-21k and apply ST-Adapter and (2) ViT-L pretrained on Clip and without ST-adapter. In this way, the reader can understand the improvement gains solely provided by the proposed ST-adapter.\n\n[1] \"Parameter-efficient transfer learning for nlp\", Houlsby et al., ICML '19\n\n[2] \"Revisiting Parameter-Efficient Tuning: Are We Really There Yet?\", Chen et al.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "pAu11FdEibf", "reviewer": "Reviewer_ZMxj", "summary": "This paper proposes a new Spatio-Temporal Adapter (ST-Adapter) for parameter-efficient fine-tuning on video tasks. With a much smaller trainable parameter, ST-Adapter can match or even outperform the strong full fine-tuning strategy on K400 and SSv2 datasets.\n", "questions": "See Strengths & Weakness.", "limitations": "No. ", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n1. It's a good attempt to introduce an Adapter for Spatio-Temporal modeling using a 2D pretrained model.\n2. The proposed ST-Adapter are effective. It can match or even outperform the strong full fine-tuning strategy with a much smaller trainable parameter.\n3. SOTA results on K400 and SSv2 using a CLIP pre-trained model.\n\nCons:\n1. Lack two important comparisons. The new added ST-Adapter has a  quite different structure from common practice for extending a 2D backbone to Spatio-temporal, e.g, an additional temporal self-attention in Figure1. So it would be better to disentangle the effect of the new structure (e.g., DW 3D-CONV which may have a complementary impact on a full Transformer architecture) and learning strategy (only tuning some new added modules, which is also called Adapter). So, exp1 [finetuning the (b) of Figure 1] and exp2 [using (a) as an Adapter of Figure 1] would be valuable.\n2. CLIP pre-trained model and ImageNet-21K pretrained model performs very differently. For example, Partial Fine-tuning using CLIP pre-trained model on K400 only performs a little worse than Full Fine-tuning (80.1 vs 81.0) while this gap is quite large when using ImageNet-21k pre-trained model (61.7 vs 76.9). It seems that the CLIP learns many video concepts which may narrow the gap between pre-training and video finetuning, especially for K400 which is largely biased to spatial appearance. So it may be more suitable to conduct an ablation study using ImageNet-21K pretrained model or using SSv2 dataset.\n3. Similar to 2, it would be helpful to add the ImageNet-22K pre-trained results in Table 2 and Table 3\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "afClvXCEhD2", "reviewer": "Reviewer_5ouG", "summary": "The paper proposes a parameter-efficient image-to-video transfer learning framework. In particular, the authors introduce a spatiotemporal adapter (ST-Adapter) for fine-tuning a pretrained image model to video tasks. In this setting, the parameters of the original network are frozen, and only the parameters of the  ST-Adapter are updated. The authors demonstrate that will only ~8% trainable parameters, they are able to achieve competitive results to previous fully fine-tuned approaches on various video recognition benchmarks. ", "questions": "See the weaknesses section above. ", "limitations": "No, the limitations were not discussed.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n+ Timely and relevant topics as video models are getting larger and larger.\n+ Strong results with a small number of trainable parameters.\n+ Detailed ablation studies for the most part. \n+ The paper is well written and easy to follow.\n\nWeaknesses:\n- The technical contribution of the paper can be seen as somewhat limited. Adapters and other parameter-efficient modules (i.e., Lora, prompt-tuning, etc.) have been widely used in other domains such as NLP, images, etc. The authors use a relatively straightforward adaptation of these ideas (with the addition of 3D depthwise convolutions, which are commonly used for efficient video processing) to the video setting. While the technical contribution is small, given that this is the first attempt (to the best of my knowledge) to perform parameter-efficient tuning on video recognition benchmarks, I'm ok with it. \n- I wish the authors validated their proposed ST-Adapter scheme with more backbones (e.g., Swin, Uniformer, Motionformer, etc.) to demonstrate its generality.\n- I think it would have also been useful to see more experiments / insights into the design choices for ST-Adapter. While 3D depthwise convolution is a natural choice, the question remains is it the best choice? Have the authors experimented with any other operators (e.g., self-attention, efficient self-attention, etc.)? In my view, it would be useful to see more design-related experiments to verify that this is indeed the adapter scheme that works most effectively.\n- Many of the Tables listing GFLOPs are unfair to the proposed method. Is it possible to highlight not only the GFLOPs but also the number of trained parameters? The proposed approach is trained using significantly fewer parameters than the remaining baselines that are fully finetuned.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "00dUfFcq0EC", "reviewer": "Reviewer_HAPR", "summary": "Authors propose an extension to the Adapter [24] framework to build a space-time adapter, which converts a pretrained image model into a video recognition model by finetuning very few parameters on the video task. The proposed adapter is very simple, based upon depthwise 3D convolution, added to each Transformer block. Experiments are shown on Kinetics and SSv2, where the proposed method obtains strong results, often comparable to fully finetuning the network (less so on SSv2, given its temporally challenging recognition task). Moreover, the resulting model is more data and training efficient.", "questions": "Please reply to the weaknesses above, especially 1 and 2.", "limitations": "N/A", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. [originality] While the method is a simple extension of \"Adapter\" [26]: The proposed method is very simple to implement and builds upon a well known Adapter architecture. Nevertheless, it obtains strong results and to my knowledge these results are not well known in the community.\n2. [quality] **Comprehensive ablations:** Authors provide comprehensive set of interesting ablations. This includes apples to apples comparisons with reasonable baselines (Table 1), as well as properties of the adapter in Fig 2 and Tab 4, which show STAdapter makes the model training and data efficient.\n3. [clarity] **The paper is generally very well written**, easy to follow, with useful and easy to read figures. \n\n## Weaknesses\n1. [quality] **Runtime savings:** While authors show impressive results in reducing parameters being finetuned yet obtaining near SOTA performance, it is unclear what the real-world advantages of such a method is. Does the reduced parameters being trained lead to speed up in training time? If so, by how much? Does that make the training possible on GPUs with less memory? If the proposed approach makes SOTA video models accessible to researchers with constrained resources, the impact of the paper will be much more.\n2. [quality] **Other backbones:** The results are largely limited to ViT based models. It's not clear if the approach can also work with other popular architectures, such as MViT and Swin? Swin for instance does come with large scale pretrained models, which would be interesting to adapt to videos.\n3. [significance] **Results without CLIP are not that strong**: Authors show impressive results, including 85.6 on Kinetics, while finetuning very few parameters and operating on very few frames from CLIP pretrained model. However, the results with ImageNet-21K are not as strong (although they do obtain improvements over other approaches for parameter efficient finetuning). It's not clear why this is the case. Perhaps more results with other \"foundation models\" (such as https://github.com/facebookresearch/SWAG) could help decipher if this approach is best applicable to vision-text models or does it work more generally. \n4. [clarity] **SOTA results missing** some higher performing methods like \"Multiview Transformers for Video Recognition\" (CVPR'22), \"Omnivore: A Single Model for Many Visual Modalities\" (CVPR'22) etc. While not directly comparable since they use different architectures and pre-training, they obtain better performance in Table 2 and 3 and should be reported when comparing to SOTA.\n5. [significance] **More datasets:** Since the proposed method is fairly simple and general, it would be nice to evaluate it more broadly for other video understanding tasks, such as egocentric action classification (eg EPIC, Ego4D), long video classification (eg HowTo100M). Such experiments would further throw light on the advantages and limitations of the proposed adapter.\n\n## Minor\n- L251: \"arts\" -> \"art\"\n- L256 \"stat\" -> \"state\"", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept