AIM: Adapting Image Models for Efficient Video Action Recognition
Authors: Taojiannan Yang
Yi Zhu
Yusheng Xie
Aston Zhang
Chen Chen
Mu Li
Venue: ICLR 2023 poster (2023)
Keywords: Video action recognition
efficient finetuning

Abstract: Recent vision transformer based video models mostly follow the ``image pre-training then finetuning" paradigm and have achieved great success on multiple video benchmarks. However, fully finetuning such a video model could be computationally expensive and unnecessary, given the pre-trained image transformer models have demonstrated exceptional transferability. In this work, we propose a novel method to Adapt pre-trained Image Models (AIM) for efficient video understanding. By freezing the pre-trained image model and adding a few lightweight Adapters, we introduce spatial adaptation, temporal adaptation and joint adaptation to gradually equip an image model with spatiotemporal reasoning capability. We show that our proposed AIM can achieve competitive or even better performance than prior arts with substantially fewer tunable parameters on four video action recognition benchmarks. Thanks to its simplicity, our method is also generally applicable to different image pre-trained models, which has the potential to leverage more powerful image foundation models in the future. The project webpage is https://adapt-image-models.github.io/.

Published as a conference paper at ICLR 2023
AIM: ADAPTING IMAGE MODELS FOR EFFICIENT VIDEO ACTION RECOGNITION
Taojiannan Yang'; Yi Zhu?, Yusheng Aston Zhang?, Chen Chen', Mu Li
"Center for Research in Computer Vision, University of Central Florida 2 AAmazon Web Services tacyangl122@knn www.ghttt.......oooooo.oooeeooee edu y yzawse amazon. com chen. chene crev. uef.edu
ABSTRACT
Recent vision transformer based video models mostly follow the "image pre- training then finetuning" paradigm and have achieved great success multiple video benchmarks. However, fully finetuning such video model could be com- putationally expensive and unnecessary, given the pre-trained image transformer models have demonstrated exceptional transferabilityy In this work, we propose a novel method Adapt pre-trained Image Models (AIM) for efficient video under- standing. By freezing the pre-trained image model and adding a few lightweight Adapters, we introduce spatial adaptation, temporal adaptation and joint adapta- tion to gradually equip image model with spatiotemporal reasoning capability. We show that our proposed AIM can achieve competitive or even better perfor- mance than prior arts with substantially fewer tunable parameters action recognition benchmarks. Thanks to its simplicity, our method is also gen- erally applicable to different image pre-trained models, which has the potential to leverage more powerful image foundation models in the future. The project webpage whttt::://// /tttss://ddpttt image models github.......................... io/
four video
INTRODUCTION
The "pre-training then finetuning" paradigm has played an important role in computer vision. The key this paradigm is well pre-trained image model, which can provide strong transferability to downstream tasks through finetuning. Recently, large foundation models (Radford et al.) [002]]
Yuan al. 2021b; Tong al. 2022 Jia et al.. 2021 Wang al.) (002bb) can even demonstrate remarkable few-/zero-shot performance given their learned superior visual representations.
In video understanding, common practice is also bootstrapping from image pre-trained model and then finetuning on the video data. There are two dominating directions as shown in Fig. one is extend an image model with additional temporal module (Lin et al., 2019) Zhu et al. [019 Arnab et al. D0 the other is to inflate an image model to a video model (Carreira & Zisserman (2017 Liu
al. 2022). >. However, there exists least two drawbacks for the aforementioned methods. First, most approaches require full finetuning (i.e., updating all the model parameters during training) to achieve promising results on common video benchmarks. This is quite costly terms of both computation and memory footprint, e.g. 1200 Tesla V100 GPU hours to train Liu et al. (2022). Second, also remains questionable that whether it is necessary to fully finetune pre-trained image models given that they have demonstrated excellent transferability. An inadequate finetuning downstream data might destroy the well generalized representations from such foundation models.
To overcome the drawbacks, a research direction termed parameter-efficient transfer learning has been trending in natural language processing (NLP) (Houlsby et al.) 2019 Lester et al. 2021 Ben Zaken al. 2022 Hu et al. 2022). The goal is only finetune small number of (extra) pa- rameters while keeping large pre-trained language models (Devlin et al., 2018 Brown et al.) 2020) frozen to attain strong performance. With the rise of large vision transformer (ViT) models, such techniques have been recently introduced to computer vision for efficient transfer learning. However, existing works either focus on tuning pre-trained image model for image tasks
(Bahng et al. 2022 Jie & Deng 2022 Jia et al. 2022), tuning a pre-trained video model for video
Work done during intersship Amazon Web Services.

---

Published as a conference paper at ICLR 2023
EVL VIFL
1400 Accursoy())
MVLIV2L
Pre-train
Tuned
MENA
Image Model
Frozen
VideoSwin-L.
83
Image Model
Image Model
TimeSfomner
Video Model
Temporal Module
Adapters
Traditional Full Finetuning
Our Efficient Tuning
10
130
Number of Tunable Parameters
160
2:0
250
570
920
Figure 1: Left: Pipeline comparison between traditional full finetuning and our efficient finetuning. Right: Performance comparison on K400 dataset (Kay et al. (207). Bubble size indicates GFLOPS at inference time. Our proposed AIM achieves the highest accuracy while enjoying significantly less number tunable parameters and GFLOPS.
tasks (video-to-video) Chen 2022 Directly leveraging pre-trained image models for efficient transfer learning to video tasks (image-to-video) is less explored, because image models lack the capability of temporal reasoning.
In this work, we introduce a new way to Adapt pre-trained Image transformer Models (AIM) for efficient video action recognition. By freezing the pre-trained image model and adding a few lightweight adapters (Houlsby et al. 2019) during finetuning, we show that our proposed AIM can achieve competitive or even better results than previous state-of-the- methods with substantially fewer tunable parameters (Fig. rightt) To be specific, we first introduce adapter after self-attention layer in a transformer block to perform spetial udaptation. We show that well pre-trained image model is sufficiently good for spatial modeling in video understanding. Then for temporal modeling. we simply reuse the image pre-trained self-attention layer but apply it to the temporal dimension of video input, forcing it to model the relationship across different frames. An adapter is also appended for temporal adaptation. Finally, we perform joint adaptation by adding another adapter in parallel to the MLP layer in transformer block. To summarize, make the following contributions:
1. We propose new way adapt pre-trained image transformer models for efficient video under- standing. Our method is generally applicable to different image pre-trained models, simple to implement, and cost-effective train.
2. Our method is significantly more efficient than fully finetuning a video model, e,g,, on Swin-B backbone, we can reduce the memory footprint by 50% and training time by 42% compared to VideoSwin (Liu et al. (0222..
AIM achieves comparable or higher performance than previous fully finetuned state-of tfite-of-the-ase the-arts 4 video action recognition benchmarks, e.g., 87.5% on K400 with 38M tunable parameters. 4. Our method also brings data efficiency, e.g., AIM outperforms counterpart TimeSformer (Berta- sius et al. 2021) by 9% absolute accuracy improvement when using 1% of the training data.
RELATED WORK
Image pre-trained models. ViT (Dosovitskiy al. 2020) and its variants (Liu et al. 2021 Wang et al. 2021b: Yuan et al. 202 Dong et al. 2022) have been proposed to achieve state-of-the-an
performance on image recognition. Once trained, these models could also serve as good initializa- tion for transfer learning to downstream tasks. In terms of training techniques, they are commonly trained on large-scale labeled datasests (Deng et al. 2009. Sun et al.] 2017]; Zhai et al., 2022) in supervised manner. To alleviate the labeling cost, self self-supervised learning methods (Chen et al.)
Bao et al. 2021 Zhou et al. 2021 He et al. 2022b Xie et al. 2022 are introduced to learn effective representations from unlabeled data. Recent works (Radford et al. 202 Jia et al. 2021 Yuan et al. 2021b Wang et al. 20226) adopt large-scale multimodal data (e.g., image-text pairs) for model training, which leads
2021
powerful visual representations. In this work, thanks to the simplicity of our proposed method, we could take advantage of these well pre-trained image
models and adapt them efficiently to solve video tasks.
2

---

Published as a conference paper at ICLR 2023
Video action recognition. A paradigm shift from using convolutional networks (Carreira & Zis- serman 2017 Tran et al. 2018 Yang et al. 2021 Lin et al.
2019 Feichtenhofer et al.) 2019) to
transtormers has been observed for video action recognition. Most works use image pre- traindd models as initialization and extend them to video models by introducing new temporal modules
(Bertasius al. 2021 Arnah et al.
202T Zhang et al. 2021b, Yan et al., 2022) or inflating them
to video models (Liu et al. (022). Another direction is to directly pre-train a vidco model in
self-supervised manner (Kuang et al. 2021 Feichtenhofer et al. 2022 Zolfaghari al. [021] Tan
et al. 2021). However, all these models are fully finetuned on video data, which makes the train- ing cost unaffordable to most researchers and practitioners. There are some recent works Ni et al.) (202)); Ju et al. )202T); Wu et al. (2023 (022) extending CLIP to perform action recognition, but they are multimodal methods which requires additional text branch. Our proposed AIM leverages existing pre-trained image models (no need for video model pre-training), only tunes small number of model parameters (much more efficient than full finetuning), and achieves comparable or even better performance than previous -fahe-fftthe-ars.
Parameter-efficien finetuning techniques (Houlsby et al., 2019 Hu et al. 2022 Lester et al. 2021 Li & Liang, 2021 He et al. 2022a; [en Zaken al. 2022 Sung et al. 202 Qing al., 2022 are first proposed in NLP since fully finetuning the increasingly larger language models for various downstream tasks becomes less feasible. Their goal is reduce the number of trainable parameters thus lowering the computation cost, while reaching or surpassing the performance of full finetuning, Recently, parameter-effiiient transfer learning is also studied in computer vision (Jia itt al. 2022; Bahng et al. 2022 Chen et al., 2022; Jie & Deng 2022 Gao et al. (022). All these methods focus on adapting models in the same domain (e.g., image-to-image or video-to-video)
while our method adapts an image model for video tasks. One concurrent work (Lin et al., (022)
also studies how adapt image traitra models for video action recognition. However, there are several major differences. First, they add new trainable decoder branches, which consist of 3D convolutions and cross-frame attention, to the frozen image encoder. We simply reuse image pre-trained self-attentio to perform temporal modeling, while enjoying better performance and less tunable parameters. Second, our method is shown to be compatible with different image models, while Lin et al. 2022) only shows its effectiveness CLIP image encoder.
3 METHODOLOGY
In this section, we first briefly describe VIT and video baselines (Sec. 3.1 Then we introduce spatial adaptation (Sec. (32)) temporal adaptation (Sec. (.3) and joint adaptation (Sec. 3.4. to show how we adapt a pre-trained image model for effective video modeling step-by-step.
3.1 PRELIMINARY
After the seminal work of Vision Transformer (ViT) (Dosovitskiy et al. 2020), transformer-based models have been widely adopted in various computer vision tasks, including video action recogni- tion. In this work, we focus on adapting pre-trained image transformer models and compare them to fully finetuned video transformer models, unless otherwise stated.
More specifically, VIT handles an image as a sequence of small patches. Given input image æ E R W'' ViT first splits the image to N non-overlapping patches and maps each patch to D- dim patch embedding via a trainable linear projection (Qian et al. 20021] 2022p. Here, (H,W) is the image resolution and C is the number of channels. Patch embeddings F. E NxD where N HW/P2 and P denotes the patch size. Then a learnable [class] token is prepended to xp as To [elass: xxp R(++))xD To encode positional information, positional embeddings E Epos E [[++1)xD are added to 2o as 20 = 2o + Epos. where zo the final input being fed to sequence of transformer blocks. Each transformer block is composed of multiheaded self-attention (MSA) and a MLP layer, together with Layernorm (LNN and skip connections, see Fig. (b). The computation of standard transformer block can be written as
2 = 21-1 + MSA (LN((L((22))))
2 2 + MLP(LN(22)),
(2)
3

---

Published as a conference paper at ICLR 2023
Tuned
MLP
Adapter
MLP
Frozen
LayorNorm
LayerNorm
MLP
MLP
1yx0
Adapter
Adapter
FC Up
LayerNorm
LayerNorm
S-MSA
S-MSA
GELU
Tx
Tx +
Adapter
LayorNorm
LayerNorm
MSA
FC Down
S-MSA
LayerNorm
[0+0)xTKO
Adapter
Adapter
LayerNorm
T-MSA
T-MSA
LayerNorm
LayerNorm
(9) Adegter
(b) VIT Block
(-) Spetial Adupistion
(d) Tempora Adeptation
(c) Joint Adeptation
We show how we adapt standard VIT block (b) for video action recognition, by gradually adding spatial adaptation (c), temporal adaptation (d) and joint adaptation (c). Note that S-MSA and T-MSA share weights but applied to different imput dimensions. During training, only newly added Adapters are updated while all the other layers are frozen.
Figure 2:
where #1-1 and zi denotes the input and output of the (-th transformer block, respectively. Finally, the learned [class] token Belass from the last transformer block is used as global visual representation and fed into a classification head to make the prediction.
Space-only and space-time models for video. A video is stack of frames with temporal struc- ture. Hence, video understanding requires the model to learn both good appearance representations in each frame (spatial modeling) and also infer the temporal information across frames (temporal modeling). In order to leverage an image transformer model for video tasks, one key thing is how to perform temporal modeling. A simple baseline, termed space-only model, process each video frame independently by an image model. Given æ E IIT xxxxHXW XC where T is the number of frames, space-only model will get T [class] tokens where each [class] token stands for the representation of each frame. These T [class] tokens will be averaged as a way of temporal modeling for final prediction. In order to enhance the capability of temporal modeling, recent works (Bertasius al., 2021 Arnab et al. 202 Zhang et al. 2021b) introduce space-time model by adding new temporal modules to image models. These models are now the top performers on most video action recogni- tion benchmarks, however, their training costs prohibitively high due to full finetuning. Given the increasingly larger and more powerful pre-trained image models, in this work, we study how to efficiently adapt them for video action recognition.
3.2 SPATIAL ADAPTATION
Since image pre-trained models have been trained on large-scale datasets and demonstrated strong transferability to downstream tasks, we believe they could achieve good spatial modeling in video action recognition with minimal finetuning.
Inspired by efficient finetuning techniques (Houlsby et al. 2019 Lester et al. 2021 Li & Liang
202 Ben Zaken al. 2022) NLP, we adopt Adapter (Houlsby al. 2019) due to its simplicity.
As shown in Fig. aa), Adapter a bottleneck architecture which consists of two fully connected (FC) layers and an activation layer in the middle. The first FC layer projects the input
lower di- mension and the second FC layer projects it back to the original dimension. To adapt the pre-trained spatial features to target video data, we add an Adapter after the self-attention layer as shown in Fig- (c.), which we term spatial adaptation. During training, all the other layers of the transformer model are frozen while only the Adapters are updated. Table [ we show that our spatial adap- tation strategy achieves comparable performance with the fully finetuned space-only baseline. This indicates that spatial adaptation helps the frozen image model to learn good spatial representations from video data. However, there is still large gap between the performance of spatial adaptation and fully finetuned video model because spatial adaptation alone lacks the ability to learn temporal information in videos.
4

---

Published as a conference paper at ICLR 2023
3.3 TEMPORAL ADAPTATION
To capture temporal information more effectively, previous methods usually incorporate new tempo- ral modules pre-trained image models because is commonly believed that image models cannot infer temporal structured information in videos. However, adding new temporal modules, either through temporal attention (Bertasius al., 2021 Zhang et al., 2021b) or temporal encoder/decoder (Amnb et al. 2021] Lin et al. (002)). will introduce sizable number of extra tunable parameters, In addition, these new modules require full finetuning, which inefficient.
To address this problem. we present new strategy: reuse the pre-trained self- self-attention layer in the image model to do temporal modeling. More specifically, we denote the original self-attention layer as S-MSA for spatial modeling, and the reused self-atentnion layer as T-MSA for temporal modeling. As shown in Fig. d),,, we put T-MSA in front of S-MSA. Now given the video patch embedding E [B'"(N-1)x we first reshape it into =" E R R[++))XTXD where N HW/ p2 is the number of spatial patches and 7 is the number of frames. Then we feed x" into the T-MSA where it tries to learn the relationship among the T frames. Note that T-MSA and S-MSA are the same layers (i.e., pre-trained MSA in the image model) and kept frozen during model tuning, but just applied to different input dimensions. This explicit operation helps our model with enhanced temporal modeling, while keeping the number of parameters fixed. In the end, similar to spatial adaptation, we add another Adapter after the reused temporal attention layer to adapt its features on video data, which term temporal adaptation. The structure of the Adapter is the same as in spatial adaptation but without the skip connection. The reason is that we want to initialize the adapted model to be close to the original model (Houlsby al. 2019), thus need to initialize the adapter to zero and remove the skip connection here to detach the effect of temporal adaptation the beginning of training. As seen in Table T finetuned video models while only introducing another lightweight Adapter into the transformer block.
temporal adaptation helps to close the gap to fully
3.4 JOINT ADAPTATION
Spatial and temporal adaptation are performed sequentially to different input dimensions with their individual purposes. It would be desirable to jointly tune the representations for spatiotemporal reasoning. To this end, we further introduce an Adapter in parallel to the MLP layer, which we term as joint adaptation. This Adapter has the same structure as the one in temporal adaptation.
The final structure of transformer block in proposed AIM is presented in Fig. 2c). The computation of the adapted block can be written as
2 -1-1 1 + Adapter
(3)
2 $ 2 + Adapter(S-MSSA(NN(T)))
(4)
= + MMP(LN(=()) + Adapter(LN(z"))
(5)
where 27. 2 2; denotes the temporal adapted, spatial adapted, and jointly adapted output in the 1-th transformer block, respectively. Here, is a scaling factor to control the weight of the output from Adapter. For the final prediction, we simply take the average of the [class] tokens of each input frame and feed the classification head.
4 EXPERIMENTS
Datasets. We evaluate the proposed method on four widely adopted video action recognition benchmarks, Kinetics-400 (K400) (Kay et al. 2017). Kinetics-700 (K700) (Carreira et al.) 2019), (SSv2) Goyal et al. 2017) and Diving-48 (Li et al. 2018) K400 contains around 240K training videos and 20K validation videos in 400 human action classes The videos all trimmed to around 10 seconds. K700 is extended version of K400 which contains around 530K training videos and 34K validation videos in 700 classes. SSv2 contains 168.9K training videos and 24.7K validation videos in 174 classes SSy2 is more challenging because it requires stronger temporal modeling (Zhu et al. 2020; (Sevilla-L Lara et al., 202T). Diving-48 contains 15.9K training videos and 2K validation videos in 48 fine-grained diving actions. is designed to be un- biased towards static representations, which means model cannot simply rely on the objects or background to determine the action.
5

---

Published as a conference paper at ICLR 2023
Table 1: Effectiveness of proposed components. We compare three baselines Something- something-v2 dataset, Spatial adaptation, temporal adaptation and joint adaptation gradually add spatiotemporal reasoning to the frozen image model. Views #frames #temporal #spatial.
Tunable
Methods
Pretrain
Param (M)
Top-1
Top-5
Views
Param (M)
Frozen space-only
IN-21K
86
0.1
15.1
36.9
8x1x3
Finetuned space-only
IN-21K
86
36.2
68.1
8x1x3
Finetuned space-time (Bertasius al., 2021
IN-21K
121
59.5
85.6
8x1x3
Frozen space-only spaltal adaptation
IN-2IK
36.7
68.3
8xIx3
+ temporal adaptation
IN-21K
10.8
61.2
87.7
8x1x3
+ joint adaptation (AIM)
IN-21K
14.3
62.0
87.9
8x1
AIM
CLIP
100
14.3
66.4
90.5
8x1x3
4.1 EFFECTIVENESS OF COMPONENTS
To demonstrate the effectivvness of our proposed components in Sec. we compare our method to three baselines. The first baseline is a frozen space-only model. Recall in Sec. 3.1 space-only model processes input frames independently and performs temporal average pooling in the end. We freeze the image backbone and only tune the classification head, which is also known as linear probing (He) al., 2020). The second baseline fully finetuned space-only model. It should be able to learn spatial information from video data, but still has difficulties in capturing temporal information. The third baseline is a fully finetuned space-time video model, which should serve as oracle. Here we choose TimeSrormer (Bertasius et al. 2021) because are based on the same ViT-B backbone and share a similar structure (i.e., divided space-time attention).
In the experiments, use the ViT-B/16 pre-trained on IN-21K as image backbone, and we compare the proposed method with the baselines on SSv2 Goyal et al. 2017) where temporal modeling is critical. The results for three baselines are shown in Tab. top. We can that the frozen space-only model only needs to tune O. IM parameters, but also performs much worse than the fully finetumed video model (15.1% 59.5%). fully finetuning the space-only model allows it learn improved spatial representations from video data and largely improves the performance (15.1% > 36.2%). However, it also significantly increases the number of tunable parameters and still has large gap from the fully finetuned video model due to lack of temporal modeling. The third baseline, fully finetuned video model, achieves the highest accuracy due its strong spatiotemporal reasoning capability, but the number of tunable parameters increases again to 121M.
Our goal is add few tunable parameters to the frozen space-only model and close the gap to fully finetuned video model. As shown in Tab. [ bottom, after spatial adaptation, the frozen space- only model achieves comparable performance with the fully finetuned space-only model (36.7% vs 36.2%), with significantly less number of tunable parameters (3.7M vs 86M). This means spatial adaptation is able to help frozen image models to achieve good spatial modeling on video data. In addition, adding temporal adaptation further boosts the performance to 61.2%, which is even higher than the fully finetuned video model. This indicates that our temporal adaptation introduces strong temporal modeling the space-only model. Finally, joint adaptation is incorporated to tune the features for improved spatiotemporal reasoning, which is our method AIM. We not only close the gap to fully finetuned space-time video model but obtain higher accuracy (62% 59.5%) with fewer number of tunable parameters (14.3M vs 86M). These results successfully validate the effectiveness of our proposed adaptation strategies.
Furthermore, our method could easily take advantage of stronger pre-trained image models and adapt them for video action recognition. For example, simply switch the ViT-B/16 pre-trained on IN-21K to CLIP pre-trained we obtain another accuracy boost (62.0% - 66.4%)
4.2 COMPARISONS TO THE STATE OF THE ART
In this section, we compare the proposed method with ofate-of-thh-an video models on four video action recognition benchmarks. For all the experiments, CLIP (Radford et al., 202T). We mostly follow the training settings in Liu et al. (2022), and more implementation details can be found in Appendix.
the ViT models mreitrinee by

---

Published as a conference paper at ICLR 2023
Table 2: Comparison state-of-the-art Kinetics-400. Views #frames x #tmmporl #spatial.
Param
Tunable Param (M)
Methods
Pretrain
GFLOPs
Top-
Top-5
Views
(M)
MVTT-B (Fan al.
2021
4095
37
37
81.2
95.1
64x3x3
UniFormer-BB B
al. 2021
IN-1K
3108
50
50
83.0
95.4
32x4x3
TimeSformer-I. Bertasius et al
2021
IN-21K
7140
121
121
80.7
94.7
64x x3
ViViT-L/I6x2 FE [Amab ct a
2021
IN-21K
3980
311
311
80.6
92.7
32x1x1
VideoSwin-LL [Lu ét
2022
IN-21K
7248
197
197
83.1
95.9
32x4x3
MV (3.7 CT ai al. 2022
IN-2IK
42420
218
218
86.1
97.0
32x3x5
MTV-L Yan al. 2022
JFT
18050
876
876
84.3
96.3
32x4x3
TokenLearner-D211 (Rvo et al.
2021
JFT
48912
450
450
85,4
96.3
64x4x3
PromptCLIP
et at (021
93.5
16x5x1
ActionCT.IP [Wang et 2121a
CL.IP
16890
142
83.8
97.1
32x10x3
X-CLIP-L/I4
2022
CLIP
7890
420
420
87.1
97.6
8x4x3
EVL ViT-L/14 (TFLL/14[imm (L
2022
CLIP
8088
368
59
87.3
32x3x1
AIM VIT-B/16
CLIP
6U6
97
83.9
96.3
8x3x1
AIM ViT-R/16
CLIP
1214
97
84.5
96.6
16x3x1
AIM VIT-B/16
CLIP
2428
97
11
84.9
96.7
32x3x1
AIM VIT-L/14
CLIP
2802
341
38
86.8
97.2
8x3x1
AIM VIT-L/14
CLIP 5604
341
38
87.3
97.6
16x3x1
AIM ViT-L/14
CLIP 11208
341
38
87.5
97.7
32x3x1
4.2.1 RESULTS ON KINETICS-400 AND KINETICS-700
Tab. 2 presents the comparisons with state-of-the-art video models K400 dataset. First, can see that with ViT-B/16 backhone, our method only needs to tune 11M parameters for competitive performance, which much smaller than previous video models. Taking input of 8 frames as an example, AIM ViT-B/16 achieves 83.9% top-1 accuracy while only requiring 606 GFLOPs. When using 16 input frames, our method even outperforms MTV-L Yaan al. 2022). 2 which requires more than 10x computations (1214 vs 18050 GFLOPs). When switching to larger backbone ViT-L/14, we achieve the highest accuracy 87.5% on K400 dataset, with 38M tunable parameters.
Note that several works also leverage CLIP pre-trained models to do video action recognition. How- ever, ActionCLII (Wang et al. 202 and X-CLIP (Ni et al., 2022) are multimodal methods which require additional text branch and tune the whole model end-to-end. PrompiCLIP (Ju et al.) [221)) applics prompt tuning (Lester et al. 2021) to CLIP and adds several temporal blocks for temporal modeling. EVL (Lin et al.) 2022) introduces new decoder branch to learn temporal information. However, AIM simply re-uses image pre-trained self-attention for temporal modeling. This makes AIM much simpler than previous methods, yet achieving better performance at much less tunable pa- rameters. The simplicity also makes AIM much easier to adapt to different model architectures (sin- gle modal multi-modal models). But previous methods such as SetiooCLIP/X-CCO wwI.CI//PPOPmCCI.. cannot leverage pure image backbone because they need an additional text branch.
Furthermore, we evaluate our method K700 dataset in Tab. 4 We can see that AIM ViT-B/16 with 11M tunable parameters is able to outperform MTV-L (875M) and MViTv2-B (51M). And
AIM ViT-L/14 (38M) achieves comparable performance with MaskFeat (218M) (Wei et al. 2022). Note that MaskFeat uses larger input resolution (312 vs 224) and more input frames (40 vs 32) than
us. This again justifies the effectiveness of our efficient adaptation pipeline.
4.2.2 RESULTS ON SOMETHING- SOMETHING-SOMETHING--
Tab. presents the performance comparisons on SSv2. Based on CLIP ViT-L/14, our method achieves competitive or better performance than most prior arts. In terms of fair comparison to EVL., which also uses CLIP pre-trained image encoder, we achieve significantly higher accuracy (70.6% > 66.7%), while introducing less tunable parameters (50M- < 175M). Note that to introduce temporal modeling into image model, EVL adds 12 layers of decoder blocks, while our method simply reuse image pre-trained self- layers to achieve stronger temporal modeling.
However, our method falls behind some fully finetuned video models (Girdhar et al. 2022 Li et al. 2022 2021). One reason is that SSy2 is cemporal-heavy. dataset (Sevilla-Larz which requires model to really understand the temporal evolution within a video. In order to obtain high accuracy, most previous video models are first pre-trained some video datasets (such as
al. 2021

---

Published as a conference paper at ICLR 2023
Table 3: Comparison to state-of-the-art on
K400 /K600' indicates the
model pre-trained on both IN-21K and K400/K600.
Param
Tunable
Methods
Pretrain
GFLOPs
Top-1
Top-5
Views
(M)
Param (M)
TimeSforer-
Bertasius al., 2021
IN-2TK
7140
121
121
62.4
64x x3
MTV-B Yan al.
2022
IN-21K
4790
310
310
67.6
90.4
32 x43
MVIT-B Fan et
2021
K400
37
37
67.1
90.8
32x1x3
MVITv2-B Li
al.
2022
K400
51
92.7
40x1x3
ViVIT-L/16 x (Amab ct u.
2021
K400
11892
311
311
65.4
89.8
16x4x3
VideoSwin-B Liu ef al. 2022
K400
963
89
89
69.6
92.7
32x1xl
Omnivore Girdhar et al. 2022
K400
71.4
93.5
x
MViTv2-L (312
2022
K400
8484
213
213
73.3
94.1
x2x1x3
UniFomer-B [Li al. 2021
K600
777
50
50
71.2
92.8
32 x
VeV (Zhang
2021a
JFT-3B
70.9
EVL Vi1-B/I6 [Ln (2022
CLIP
2047
182
86
62.4
32x1x3
EVL ViT-L/14
Lin al.
2022
CLIP
9641
484
175
66.7
xx31x3
AIM VTT-B/16
CLIP
624
100
14
66.4
90.5
8x1
AIM ViT-B/16
CLIP
1248
100
14
68.1
91.8
16x1x3
AIM ViT-B/16
CL.IP
2496
100
14
69.1
92.2
x
AIM ViT-L/14
CLIP
2877
354
50
67.6
91.6
8x1x3
AIM ViT-L/14
CLIP
5754
354
50
69.4
92.3
16x 16x1x3
AIM ViT-L/14
CLIP
11508
354
50
70.6
92.7
32x1x3
Table Comparisons on Kinetics-7000
Table 5: Comparisons on Diving-48.
Tunable
Tunable
Top-1
Method
Pretrain
Param
Top-1
VHITR-L Zhame et al 2221
IN-2TK
91
702
TimeSformer- (Bertasins et
2021
IN-21K
121
81.0
MTV-L
IN-21K
876
75.2
VidcoSwin (Lau
IN-21K
MViTv2-B
2022
BEVT Wang al Pizza
K4001
88
86.7
MViTvZ-L. 310
Trii al 2023
IN-21K
218
79.4
SIFAR-
2022
IN-21K
87
87.3
FaskFreat (40 x 312
Wer at 2012
K700
218
20.4
Herzig er
IN-21K
AIM WT-B/16
CLI
11
76.9
AIM Vi--8/T
CLIP
11
88.9
AIM VIF-L/14
CLIP
38
90.6
K400/K600) to learn good spatiotemporal representations, then finetuned on SSv2. But method still starts from the image pre-trained model. Another reason is that simply reusing the image pre- trained self-attention for temporal modeling may not be able fully capture the complicated tem- poral information in SSv2 videos. This suggests that we need to conduct more temporal adaptation for these challenging "'emporal-heavy" datasets.
4.2.3 RESULTS ON DIVING-48
A diving class in Diving-48 (Li et al, 2018) is defined by the combination of takeoff, movements in
flight and entry, thus it requires the model to differentiate such fine-grained actions. As shown in Tab. [] our method with IM tunable parameters outperforms all prior methods. AIM ViT-L/14 further improves the top-1 accuracy to 90.6%. Comparing ORViT (Herzig leverage additional object tracking model, our method still outperforms it with much less tunable parameters. This suggests that efficient finetuning can handle fin-ggrained action recognition.
2022), despite they
DISCUSSION
Different Pre-trained Models. Here demonstrate the effectiveness of AIM on dif-
Table Performance using different pre- trained models K400.
ferent pre-trained models. In Table 6 we first
show AIM based on ViT-B backbone. We
Param
(H)
Top-1
compare AIM to TimeSformer because we use the same backbone (ViT-B) and have similar structure (i.e., both using divided space-time at- tention). As can be seen, AIM achieves better performance than fully finetuned TimeSformer
TimeStommer AIM
ViT-B
IN-IIK
131
10
20
78.5
VIT-B
IN-21K
11
15
78.8
TimeSformer
VITB
CLIP
20
82.0
VideoSwin-B
Swin-B
IN2IX
61
AIM
Swin-B
IN-2KK
9.2
37
82.1
under both IN-21K and CLIP pre-trained weights. Then apply AIM to Swin-B backbone and compare it to VideoSwin when we both use Swin-B and IN-21K pre-training. Similarly, AIM achieves comparable performance with fully finetuned VideoSwin.
8

---

Published as a conference paper at ICLR 2023
83.9
Table 7: Effect of position of Adapters. Skip
means adding Adapters every two blocks.
79.3
75
73.0
Position
Tunable
Top-1
K405 Topal Accuacy
Param (M)
69.5
Bottom 6
5.6
80.7
Top 6
5.6
83.3
65
Skip All
5.6
83.2
83.9
55
Table 8: Effect of bottleneck ratio of Adapters.
AM Yandline
Tunable Param (M)
45
46.5
Ratio
Top-1
Amount of Training Data (lgg scale)
0.0625
3
83.3
Figure 3: Data efficiency comparison. AIM out- performs fully finetuned TimeSformer under all scenarios, especially in low data regime.
0,
5.6
83,4
0.25
11
83.9
0.5
21
83.8
Data Efficiency. One advantage of our efficient tuning paradigm is that we can keep the well pre-trained image representations intact. the scenario where downtream data is insufficient, our method will be less prone to over-fitting compared to full finetuning. In Fig. 3 we compare AIM with fully finetuned TimeSformer under different amounts of training data on K400. For fair com- parison, both AIM and TimeSformer use CLIP pre-trained ViT-B/16 backbone. We can observe that under all scenarios, our method AIM outperforms fully finetuned TimeSformer. In particular, when the amount of data becomes less, the advantage of AIM becomes larger. For example, when there is only 1 1% of training data, we outperform TimeSformer by a significant margin of 8.9%.
Training Cost. Tab. Galso shows the training time (hours) and memory cost (GB) of our method and fully finetuning on different backbones. All metrics are measured on 8 Tesla V100 GPUs. Compared to TimeSformer,
reduce the memory cost by 30% and training time by 25%. Compared to VideoSwin, we reduce the memory cost by 50% and training time by 42%.
Position of Adapters. By default, we add Adapters every ViT block (12 blocks in total). Here we study the effect of adding Adapters in different layers. We add Adapters to the bottom 6 blocks (close to the input), top blocks (close to the output) and one every two blocks. All these variants have the same number of tunable parameters. As can be seen in Tab. 7 adding Adapters to the bottom blocks yields much worse performance than others. We hypothesize that the shallow layers learn generic representations which do not need much adaptation, while deeper layers learn task-specific features like temporal information thus feature adaptation is important. Adding Adapters to the top blocks achieves comparable performance with adding to all blocks while saving half of the parameters. This could serve as a good candidate when training resources are more limited.
Bottleneck Ratio of Adapters. the number of tunable parameters. Here we study how the bottleneck ratio of Adapters affects the final performance. The results in Tab. g reveal that larger bottleneck ratio tends to achieve better performance, but it will also introduce more tunable parameters. The performance plateaus after bottleneck ratio goes beyond 0.25. Note that small ratio of 0.0625 could still achieve 83.3% top-1 accuracy on K400, which is competitive among state-of-the-art video models in Tab. 2 while introducing only 3M tunable parameters.
By tuning the bottleneck ratio of Adapters, can easily control
CONCLUSION
In this work, we propose new way to efficiently transfer pre-trained image models for video action recognition. We introduce spatial adaptation, temporal adaptation and joint adaptation to gradually add spatiotemporal reasoning to an image model. Since only newly added Adapters are updated, our training cost is substantially lower than other fully finetuned video models. Yet we achieve com- parable or even better performance than prior arts on four benchmarks. Our method is simple and generally applicable, which has the potential to leverage more powerful image foundation models in the future. Despite all the benefits, one limitation is that our simple strategy of reusing spatial at- tention for temporal modeling might not be strong enough for temporally challenging videos. Since video temporal modeling can be viewed as a form of sequence modeling, we might be able to reuse pre-trained weights from text or audio models instead of image models in the future.

---

Published as a conference paper at ICLR 2023
REFERENCES
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6836-6846, 2021.
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Visual prompting: Modi- fying pixel space to adapt pre-trained models. arXiv preprint arXi::2203.17274, 2022.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BEiT: BERT Pre-Training of Image Trans- formers. arXiv preprint arXiv:2106.082:: 2021.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. BitFit: Simple parameter-efficient fine- tuning for transformer-basee masked language-modele In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 1-9, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10. 10.18663//1/2022..a0 short. /ttp::://aalaaiie peattps://aclantholoo org
1. URL
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In Proceedings of the International Conference on Machine Learning (ICML), July 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33: 33:1877-1901, 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299-6300 2017.
Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset. arXiv preprint (rxxiv:/07.000 2019.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. arXiv preprint arXiv:2205.13535, 2022.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self supervueee vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 9640-9649, 2021.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition workshops, pp. 702-703, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. 2009 IEEE Conference on Computer Vision and Puttern Recognition, pp. 248-255. IEEE, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.044: 04805, 2018.
Xiaoyi Dong, Jianmin Bao, Dongdong Chen, Weiming Zhang, Nenghai Yu, Lu Yuan, Dong Chen, and Baining Guo. Cswin transformer: A general vision transformer backbone with cross-shaped windows. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition, pp. 12124-12133 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proceedings of the IEEE/CVF Inter- national Conference Computer Vision, pp. 6824-6835, 2021.
10

---

Published as a conference paper at ICLR 2023
Quanfu Fan, Chun-Fu Chen, and Rameswar Panda. Can image classifier suffice for action recognition? In International Conference on Learning Representations, 2022. URL https:
//openzeview.nee wwwpeeeeeorndaaaaaia.... HH
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp- 6202-6211, 2019.
Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders spa- tiotemporal learners. arXiv preprint arXiv:2205.091133 2022.
Yunhe Gao, Xingjian Shi, Yi Zhu, Hao Wang, Zhiqiang Tang, Xiong Zhou, Mu Li, and Dim- itris N. Metaxas. Visual Prompt Tuning for Test-time Domain Adaptation. arXiv preprint DrXiv:2210004831, 2022.
Rohit Girdhar, Mannat Singh, Nikhila Ravi, Laurens van der Maaten, Armand Joulin, and Ishan Misra. Omnivore: A single model for many visual modalities. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 16102-16112 2022.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Hcuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag. et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3842-5850, 2017.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kinn-KKikaattick, and Graham Neubig. Towards a unified view of parameter-efficien transfer learning. In International Conference on Learning Representations, 2022a. URL httptttptt//peeeeeiiiiiie htt..eeeeetooooo222ooooococa.o.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9729-9722 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022b.
Roei Herzig, Elad Ben-Avraham, Karttikeya Mangalam, Amir Bar, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Object-region video transformers. In Proceedings of the JEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 3148-3159, 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799. PMLR, 2019.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con-
ference on Learning Representations, 2022. URL htttss:/ weteeiiwwwweeeeeeeeeeeeeeeeeeeee net wwt.:ee//ooroooo fformm?
id=nZeVKeeFYf9
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Scr-Nam Lim. Visual prompt tuning, arXiv preprint 1:XXi:2203.121199 2022.
Shibo Jie and Zhi-Hong Deng. Convolutional bypasses are better vision transformer adapters. arXiv preprint ar 220:22077007339 2022.
Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. arXiv preprint arXiv:21/2.04478 2021.
11

---

Published as a conference paper at ICLR 2023
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950, 2017.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141.6::412.6980, 2014.
Haofei Kuang, Yi Zhu, Zhi Zhang, Xinyu Li, Joseph Tighe, Sören Schwertfeger, Cyrill Stachniss. and Mu Li. Video contrastive learning with global context. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 3195-32000 2021.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104.08691, 2021.
Kunchang Li, Yali Wang, Gao Peng, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unified transformer for efficient spatial- temporal representation learning. In International Con- ference Learning Representations, 2021.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 1582-4597, Online, August 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021 aat.....c--ooo3555 URL https: http://aaaaatthhooco wttp:/hhooooooo..o21o0....11 attp:///2222..a11ooooo.o.
Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4794-4804, 2022. doi: 10.1 1091////22552888... SWW.CVVPR2268.222200007.7.
Yingwei Li, Yi and Nuno Vasconcelos. Resound: Towards action recognition without representa- tion bias. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 513-528, 2018.
Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 7083-7093, 2019.
Ziyi Lin, Shijie Geng, Renrui Zhang, Peng Gao, Gerard de Melo, Xiaogang Wang. Jifeng Dai, Yu Qiao, and Hongsheng Li. Frozen clip models are efficient video learners. arXiv preprint arXiv:2208. 03550, 2022.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision pp. 10012-10022, 2021.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin trans- former. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- fion, pp. 3202-3211, 1,, 2022.
Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for general video recog- nition. arXiv preprint arXiv:2208.022. 02, 2022.
Shengju Qian, Hao Shao, Yi Zhu, Mu Li, and Jiaya Jia. Blending anti-aliasing into vision trans- former. Advances in Neural Information Processing Systems, 34:5416-5429, 2021.
Shengju Qian, Yi Zhu, Wenbo Li, Mu Li, and Jiaya Jia. What makes for good tokenizers in vision transformer? IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
Zhiwu Qing, Shiwei Zhang, Ziyuan Huang, Xiang Wang, Yuehuan Wang, Yiliang Lv, Changxin Gao, and Nong Sang. Mar: Masked autoencoders for efficient action recognition. arXiv preprint arXiv:2207. 1660, 2022.
12

---

Published as a conference paper at ICLR 2023
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Michael S Ryoo, AJ Piergiovanni, Anurag Arnab, Mostafa Dehghani, and Anelia Angelova. Token- learner: Adaptive space-time tokenization for videos. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances Neural Information Processing Systems, 2021. URL, https::/
net forum?id=ze (orrmmiid=-IIIppDSs- 11kpDXs88
Laura Sevilla-Lara, Shengxin Zha, Zhicheng Yan, Vedanuj Goswami, Matt Feiszli, and Lorenzo Torresani. Only time can tell: Discovering temporal data for temporal modeling. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 535-544, 2021.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable ef- fectiveness of data in deep learning era. In Proceedings of the IEEE international conference on computer vision, pp. 843-852,20 2017.
Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. Advances in Neural Information Processing Systems, 34:24193-24205, 2021.
Hao Tan, Jie Lei, Thomas Wolf, and Mohit Bansal. Vimpac: Video pre-training via masked token prediction and contrastive learning. arXiv preprint arXiv:2106. 1250, 2021.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang- Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training arXiv preprint urXiv:2203.12600 2022.
Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. closer look spatiotemporal convolutions for action recognition. In Proceedings of the IEEE Confer- ence on Computer Vision and Pattern Recognition, pp. 6450-6459, 2018.
Mengmeng Wang, Jiazheng Xing, and Yong Liu. Actionclip: A new paradigm for video action recognition. arXiv preprint arXiv:2109.08472 2021a.
Rui Wang, Dongdong Chen, Zuxuan Wu, Yinpeng Chen, Xiyang Dai, Mengchen Liu, Yu-Gang Jiang, Luowei Zhou, and Lu Yuan. Bevt: Bert pretraining of video transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14733-14743, 2022a.
Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao. Pyramid vision transformer: versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 568-578, 2021b.
Wenhui Wang et al. Image as Foreign Language: BEiT Pretraining for All Vision and Vision- Language Tasks. arXiv preprint arXiv:2208, 10442, 2022b.
Chen Wei, Haoqi Fan, Saining Xie, Chao-Yus Wu, Alan Yuille, and Christoph Feichten- hofer. Masked feature prediction for self-supervised visual pre-training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14668-14678, 2022.
Wenhao Wu, Xiaohan Wang, Haipeng Luo, Jingdong Wang, Yi Yang, and Wanli Ouyang. Bidirec- tional cross-modal knowledge exploration for video recognition with pre-trained vision-language models. arXiv preprint arXiv:2301.0m 00182, 2022.
WenHao WU, Zhun Sun, and Wanli Ouyang. Revisiting classifier: Transferring vision-language models for video recognition. 2023.
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9653-96633 2022.
13

---

Published as a conference paper at ICLR 2023
Shen Yan, Xuehan Xiong, Anurag Arnb, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3333-3345 2022.
Taojiannan Yang. Sijie Zhu, Matias Mendieta, Pu Wang. Ravikumar Balakrishnan, Minwoo Lee, Tao Han, Mubarak Shah, and Chen Chen. Mutualnet: Adaptive convnet mutual learning from different model configurations. IEEE Transactions on Patlern Analysis and Machine Intelligence, 45(1):811 1-87 2021.
Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. In International Conference on Learning Reprexentations, 2021.
Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan. Tokens-to-token vit: Training vision transformers from scratch on imagenet. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 558-567, 2021a.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, al. Florence: A new foundation model for computer vision. arXiv preprint vrXiv:2111.11432. 2021b.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling Vision Transformers. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
Bowen Zhang, Jiahui Yu, Christopher Fifty, Wei Han, Andrew M Dai, Ruoming Pang, and Fei Sha. Co-training transformer with videos and images improves action recognition. arXiv preprint 1rXiv:2112.07175, 2021a
Yanyi Zhang, Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Mar- sic, and Joseph Tighe. Vidtr: Video transformer without convolutions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 13577-13587, 2021b.
Haodong Zhao, Wei Du, Fangqi Li, Peixuan Li, and Gongshen Liu. Reduce communication costs and preserve privacy: Prompt tuning method in federated learning. arXiv preprint arXiv:2208. /2268, 2022.
Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yi Yang. Random erasing data augmen- tation. In Proceedings of the AAAI conference artificial intelligence, volume 34, pp. 13001- 13008, 2020.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint 2rXtv:2111.07832,20 2021.
Yi Zhu, Zhenzhong Lan, Shawn Newsam, and Alexander Hauptmann. Hidden two-stream convolu- tional networks for action recognition. In Computer Vision-ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2-6, 2018, Revised Selected Papers, Part III 14, pp. 363-378. Springer, 2019.
Yi Zhu, Xinyu Li, Chunhui Liu, Mohammadreza Zolfaghari, Yuanjun Xiong, Chongruo Wu, Zhi Zhang, Joseph Tighe, R. Manmatha, and Mu Li. A comprehensive study of deep video action recognition. arXiv preprint arXiv: 2020.
Mohammadreza Zolfaghari, Yi Zhu, Peter Gehler, and Thomas Brox. Crossclr: Cross-modal con- trastive learning for multi-modal video representations. Proceedings of the IEEE/CVF Inter- national Conference Computer Vision, pp. 1450-1459, 2021.
14

---

Published as a conference paper at ICLR 2023
A IMPLEMENTATION DETAILS
A.1 KINETICS 400/700
The bottleneck 0.25 and the scaling factor is set to 0.5. The first FC layer in Adapters randomly initialized and the second FC layer is initialized to zero. In this way, the adapted model close to the pre-trained model the beginning of training. We largely follow the training settings and data augmentations in Liu et al. (0022). Specifically, the model is trained for 30 epochs using AdamW (Kingma & Ba, 2014) optimizer with a batchsize of 64. The base learning rate 3e-4 and weight decay is 5e-2. The learning rate is warmed up from in the first 3 epochs and then decays following a cosine schedule. The stochastic depth rate is 0.2 for both ViT-B and ViT-L.. For inference, we sample three clips along the temporal dimension. The final performance is evaluated by the ensemble of three views. We evaluate the model on 8, 16, frames and the sampling interval is 16, 8, 4, respectively.
We add spatial/temporaljoint adaptation in every VIT block as shown in Fig. 2
ratios of all adapters
A.2 SOMETHING-SOMETHINGGE
We add spatial/tepporaljoontt adaptation in every ViT block as shown Fig. 2 We additionally add one adapter before T-MSA to enhance the temporal modeling. The bottleneck ratios of all adapters are set to 0.25 and the scaling factor is set to 0.5. We follow Liu et al. augmentations including label smoothing, RandAugment (Cubuk et al., 2020) and random erasing (Zhong et al., 2020). The model is trained for 50 epochs using Adam (Kingma & Ba. 2014) optimizer. The other training settings are the same Kinetics-400. We uniformly sample 8, 16, 32 frames the experiments. For inference, we sample three spatial crops. The final performance is evaluated by the ensemble of three views.
(202)) to use stronger data
A.3 DIVING-48
We add spatial/temporaljoonnt adaptation in every VIT block as shown in Fig. 2 The bottleneck ratios of all adapters are set to 0.25 and the scaling factor is set to 0.5. The model is trained for 50 epochs. The other training settings and data augmentations are the same as K400. We uniformly sample 8, 16, 32 frames in the experiments. For inference, we only sample I temporal clip.
B VISUALIZATION
In this section, present the attention map visualizations of the frozen space-only model, Spatial Adaptation (SA) model, Spatial Adaptation plus Temporal Adaptation (TA) model, and the fully finetuned TimeSformer.
On Fig. 4 left, we visualize an action "Brush Painting" from Kinetics-400 dataset. We can see that the attention maps of the frozen space-only model are very scattered, and doesn't attend to the brush region in the first two frames. Adding SA enhances the attention on the brush, but the model still focuses on areas that are unrelated to the action. Further adding TA helps the model learn temporal information. We can see that the model now focuses more on the brush painting area, which is similar to what fully finetuned TimeSformer does.
On Fig. righht we visualize an action "Something falling like a rock" from Bomething-Somethiing v2 dataset. To correctly recognize this action, the model needs to learn how the object moves in the input frames. We first observe that both the frozen space-only model and SA model have good attention on the object, but they fail to model the movement of the object which leads to wrong prediction. In contrast, TA helps the model to leam the relationship among input frames. The attention map shows that the model not only focuses on the object but also learns the track of the object. Instead, TimeSformer always attends to the bottom region without showing the object path.
15

---

Published as a conference paper at ICLR 2023
Label: Brush Painting
Label: Something falling like a rock
Triss
wwvespspedyyy SAATA
Figure 4: Attention map visualizations of AIM variants and the fully finetuned TimeSformer. With the help of temporal adaptation (TA), our method is able focus motion salient regions which helps to make a correct prediction.
Spilling something behind something
Pretending tum something upside down
+23
Poking stack of something the stack collapses
+20
Putting something sur face
+19
Pilling something
119
-23
Turning right while filming something
-27
Pushing something from to right
-31
Pulling something from left right
-32
Pulling something right
-43
Turning the camera left while filming something
-50
-40
-30
-20
-10
10
20
30
40
A Class Accuracy (%) AIM vs. TimeSfermer
Figure 5: The figure shows the differences of each class's accuracy of AIM and TimeSformer on v2. Here only plot the top-5 and bottom-5 classes.
C PER-CLASS ANALYSIS
In Tab. we show that AIM still falls behind some SoTA fully finetuned video models on the (SSv2) dataset. We conjecture one reason that simply
"temporal-heavy" Something-SSmmttinngg reusing the image pre-trained self-attention for temporal modeling may not be able to fully capture the complicated temporal information in some nuanced action classes SSv2. To provide further insights. compute the per-class accuracy differences of AIM and TimeSformer SSv2 and show the t and bottom-5 classes in Fig. []] We can see that the classes where AIM performs better are normal action classes with decent motion. The classes where ATM performs worse those with
16

---

Published as a conference paper at ICLR 2023
Table 9: Comparisons of the training memory cost of AIM and fully finetuned models based on large
image pre-trained backbones. AIM significantly reduces the memory cost and makes large model training easier.
Model
Backbone
Mem (G)
TimeSformer Bertasius Bertassius// al. (2021)
ViT-L
21.2
AIM
ViT-L
14.3
VideoSwin Liu et (2022)
Swin-L.
Out of Memory
AIM
Swin-I.
13.7
Table 10: Comparisons with EVL under different pre-trained datasets. AIM outperforms EVL under
different pre- praitrann and uses less number of tunable parameters.
Tunable
Model
Backbone
Pretrain
Mem (G)
Time (H)
Top-1
Param (M)
EVL Lin et al.
(0022)
ViT-B
IN-21K
36.3
4.2
29
75.4
AIM
VYT-B
IN-21K
11
7
15
78.8
EVL Lin et al. (2022)
VIT-B
CLIP
36.3
4.2
29
82.9
AIM
VIT-B
CLIP
11
7
15
83.9
minor differences (e.g., "Pulling something from left to right" vs. "Pulling something from right to left"). In order to tell these actions apart, the model needs to distinguish between the nuances, especially in motion. Given most model parameters are frozen in our method, AIM may lack the capacity capture such complex temporal information.
D MORE COMPARISONS TRAINING COST
In T demonstrate the training efficiency of AIM based on VIT-B and Swin-B backbones. In this section, we show more comparisons with fully finetuned baselines based on ViT-L Swin-L backbones. The results are shown in Table 9 We can see that TimeSformer with VIT-L backbone needs 21.2G GPU memory, and VideSwin with a Swin-L backbone cannot fit into an 8 Tesla V100 32G GPU server. In both cases, AIM can significantly reduce the memory usage to 14.3G and 13.7G, respectively. This makes large model training more memory-friende (runnable on most GPUs with 15G memory and more). and thus more affordable for most researchers and practioners.
Furthermore, beyond memory saving, optimizing number of tunable parameters has potential bene- fits in other applications such communication-efficie distributed learning (e.g., federated learn- ing where the tunable model parameters are communicated between the central server and local clients) and privacy preserving federated learning Zhao al. (2022); y; Yu al. (202T). Tuning less parameters could also be beneficial when the downstream data is limited because fully finetuning a large model on limited data may suffer from serious overfitting. This can be observed from the results in Fig. where AIM obtains larger accuracy improvements over the fully finetuned baseline when there only small amount training data.
COMPARISON TO EVL UNDER DIFFERENT PRE-TRAING DATASETS
In this section, we compare AIM with EVL Lin 2022). which is the most recent SoTA image- to-video efficient finetuning method based on frozen pre-trained VIT. As shown in the Table 10 AIM consistently outperforms EVL under both IN-21K K and CLIP pre-training as well. And AIM uses considerably less tunable number of parameters than EVL.
F PSEUDO-CODE OF THE ADAPTED VIT BLOCK
As explained in the paper, AIM is effective and simple to implement. Algorithm we show the PyTorch style pseudo-code how apply AIM to VIT block.
17

---

Published as a conference paper at ICLR 2023
Pseudo-code of an adapted VIT block class Transformerßlock/ ()
Algorithm
def
init (self, dim, num head, mip_ ratio, scale)
## Layers in the original Vir block
self.attn hulttheadkttention (dim, num head)
self.norml LayerNorm (dim)
sclf.mlp MLP (dim, mlp_ratio) sclf.norm2 LayerNorm (dim)
# Adapters
self self.s_adapter Adapter (dim)
seli.t_adapter Adapter (dim)
selt.mip_adapter adapter = Adapter (dim)
self.scale scale
def forward
#8 x in shape [N+1, BT, D]
## temporal adaptation
xt rearrange (x, (b
n) d', t-nur_frammee
xt celf
(x))
xt rearrange (x, 't (b n) d n (b t) d', a=num_patches)
x x xt
spatial adaptation
x x self self.s_adapter (st (self. norml )x))
## joint adaptation
self. www...orm2leeeeemmmmmm...meem (x)
self.mlp (x_norm) self self.mlp_adapter (X_norm)
return
18


Reviews:
[{"review_id": "iqfeqZbIPCw", "reviewer": "Reviewer_jsrf", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes AIM (Adapting Image Models) for video classification with the focus on reducing tunable parameter and training (fine-tuning) time. The main contributions are: 1) using Adapter (Houlsby et al. 2019) to adapt image models (pretrained on large datasets such as ImageNet-21k or CLIP); 2) designing various spatial, temporal, and joint adaptation blocks. The paper provides strong ablation experiments and solid results compared with state-of-the-art (SoTA) methods on multiple benchmarks including K-400, K700, SSv2, and Diving-48. Written presentation is clear and easy to read.", "summary_of_the_review": "In summary, the paper is novel and interesting, experiments are solid and potential impactful (useful for academic labs). I recommend to accept this paper.", "strength_and_weaknesses": "# Strength\n- The idea of adapting image models (pretrained on large benchmarks) to video classification with minimum numbers of tunable parameters and training time is novel and interesting (not truly novel due to Lin et al. 2022, but the authors clearly state the differences).\n- AIM showed many advantages compared with SoTA methods where full-fine-tuning is used. AIM is particularly efficient in terms of number of tunable parameters, training time, and data efficiency. In most cases, we observe AIM provides better performances (except for SSv2).\n- Ablation experiments are enough to understand the design choice.\n- The trick that using weight from spatial-attention for temporal attention (then keep them frozen) is nice. In deep learning we saw this trick many times (e.g., CDC or inflation of image models) but most of the time, it is followed by full-fine-tuning. AIM re-purpose the spatial weights for temporal task, w/o full-fine-tuning, instead adapting them. This makes it quite interesting.\n\n# Weakness\n- Not a major weakness, but the paper titled as \"video understanding\", it is more suitable to make it \"video classification\" or \"video recognition\" since all benchmarks and tasks experimented in the paper are classification.   ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "gX6SyyarRH", "reviewer": "Reviewer_auyL", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an efficient method of transferring image pre-trained weights to video processing. In summary, the paper proposes to freeze most of the pre-trained model parameters with only a small portion of parameters in the newly inserted Adapter module learnable. The authors have show the effectiveness of the proposed AIM methods in several datasets and significant improvements over the selected baselines.", "summary_of_the_review": "Overall, the paper is well motivated and the proposed solution is reasonable. However, due to some missing ablations, I keep my score as boardline (leaning accept). I will change my score accordingly based on the author's response.", "strength_and_weaknesses": "Strength:\n1. The paper is well motivated. I agree that the well pre-trained model weights are not necessary to be fine-tuned for adapting to video tasks.\n2. The paper is well written and easy to follow.\n3. The proposed methods are parameter efficient based on the presented experiments.\n4. With the adapters inserted, even with less number of tunable parameters and memory cost, AIM achieves higher accuracy than ViT baseline method. Could the authors explain why this could happen?\n\nWeakness:\n1. The efficiency is not clear in terms of memory consumption, latency and throughput. The discussion on the saving on the memory and training hours are reported separately in Tab. 6 in the paper. I would suggest to put the memory and latency metrics in Tab. 3 also to better reflect the advantages over baseline methods.\n2. The ablations on the impact of AIM over traditional full tuning is not solid. It would be clearer to see the impact of adapter by comparing the AIM to traditional full tuning method. Currently, most of the method selected in Tab.3, Tab. 4 and Tab. 5 are not pre-trained on CLIP, while AIM methods are all pre-trained on CLIP. Then, it is hard to check, how much benefits come from AIM and how much of the benefits come from CLIP pre-training. Although this is explored separately in Tab. 6, the comparison with baselines models are missing.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VXjP2PhmXt", "reviewer": "Reviewer_UEFx", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a simple way for adapting image pretrained models to videos. Specifically, authors leverage the default ViT architecture, add a temporal attention layer simply by using the same weights learned for spatial attention, and add lightweight `adapter' layers that consists of a bottleneck block. Only the adapter layers are trained, and they are added after the spatial attention, temporal attention, and as a residual to the last frozen MLP in the transformer block. The resulting model obtains strong results on 4 benchmark, closely matching SOTA. The resulting model is also data efficient and gets larger gains with small amounts of training data (Fig 3).", "summary_of_the_review": "I am very borderline on this paper. I'm leaning towards accept since this is a useful technique (initializing from image models), solves an important problem (video recognition), and is perhaps the simplest instantiation of a solution. ", "strength_and_weaknesses": "## Strengths\n1. Simple and effective approach and doesn't need the text trunk: Perhaps the strongest part about this paper is the simplicity of the approach. Even compared to recent work like EVL (Lin et al. 2022), which trains a decoder to combine frames, and ActionCLIP/XCLIP/CLIP-ViP ((https://arxiv.org/abs/2209.06430) which leverage the text trunk as well, this method simply uses the visual trunk only and is able to adapt it for supervised classification training without needing any text prompting. It simply leverages the adapter architecture proposed in NLP ((Houlsby et al., 2019).\n\n2. Shows gains in data efficient setup: The proposed approach seems particularly effective in few-shot settings. It is however unclear how it compares to few-shot performance of the other contemporary CLIP->video methods (as in point 1 above).\n\n## Weaknesses\n1. Somewhat weak results: As noted in strength point 1, CLIP->video models is a very competitive space. Although the method is simpler, the performance isn't particularly better than other approaches as noted in most of the tables. Also the performance does lag fully supervised end-to-end trained methods like Omnivore and MVIT-v2 on challenging temporal reasoning datasets like SSv2 (Table 3), (Omnivore/MVIT don't even use CLIP features). This suggests the approach may not scale well towards challenging temporal reasoning video tasks and is probably best suited for Kinetics-like tasks (which anyway works well even with the original frame-level CLIP features).\n\n2. Compute savings aren't huge: Another analysis/aspect I liked about the method was compute analysis in Table 6. However as the numbers show, the huge gains in parameter savings don't really transfer to compute/train-time/memory savings as much. Since the trainable parameter count isn't a particularly useful metric to optimize for, it's not clear if users/practitioners are indeed better off with adding these additional adapter layers, or should they just finetune the full model without modifying the architecture at a bit higher runtime cost (30% in case of ViT), but a near guarantee of better performance.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster