Towards a Unified View on Visual Parameter-Efficient Transfer Learning
Authors: Bruce Yu
Jianlong Chang
Lingbo Liu
Qi Tian
Chang Wen Chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Parameter Efficient
Transfer Learning
Domain Adaption

Abstract: Since the release of various large-scale natural language processing (NLP) pre-trained models, parameter efficient transfer learning (PETL) has become a popular paradigm capable of achieving impressive performance on various downstream tasks. PETL aims at making good use of the representation knowledge in the pre-trained large models by fine-tuning a small number of parameters. Recently, it has also attracted increasing attention to developing various PETL techniques for vision tasks. Popular PETL techniques such as Prompt Tuning and Adapter have been proposed for high-level visual downstream tasks such as image classification and video recognition. However, Prefix-tuning remains under-explored for vision tasks. In this work, we intend to adapt large video-based models to downstream tasks with a good parameter-accuracy trade-off. Towards this goal, we propose a framework with a unified view of PETL called visual-PETL (V-PETL) to investigate the effects of different PETL techniques, data scales of downstream domains, positions of trainable parameters, and other aspects affecting the trade-off. 
Specifically, we analyze the positional importance of trainable parameters and the differences between NLP and vision tasks in terms of data structures and pre-training mechanisms while implementing various PETL techniques, especially for the under-explored prefix-tuning technique. Based on a comprehensive understanding of the differences between NLP and video data, we propose a new variation of prefix-tuning module called parallel attention (PATT) for video-based downstream tasks. 
An extensive empirical analysis on two video datasets via different frozen backbones has been carried and the findings show that the proposed PATT can effectively contribute to other PETL techniques. An effective scheme Swin-BAPAT derived from the proposed V-PETL framework achieves significantly better performance than the state-of-the-art AdaptFormer-Swin with slightly more parameters and outperforms full-tuning with far less parameters.

Under review as a conference paper at ICLR 2023
TOWARDS A UNIFIED VIEW ON VISUAL PARAMETER- EFFICIENT TRANSFER LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
Since the release of various large-scale natural language processing (NLP) pre- trained models, parameter efficient transfer learning (PETL) has become popu- lar paradigm capable of achieving impressive performance on various downstream tasks. PETI, aims at making good use of the representation knowledge in the pre- trained large models by fine-tuning small number of parameters. Recently, it has also attracted increasing attention to developing various PETL techniques for vision tasks. Popular PETL techniques such as Prompt-tuning and Adapter have been proposed for high-level visual downstream tasks such as image classifica- tion and video recognition. However, Prefix-tuning remains under-explored for vision tasks. In this work, we intend to adapt large video-based models to down- stream tasks with good parameter-accuracy trade-off. Towards this goal, we propose a framework with unified view of PETL called visual-PETL. (V-PETL) investigate the effects of different PETL techniques, data scales of downstream domains, positions of trainable parameters, and other aspects affecting the trade- off. Specifically, we analyze the positional importance of trainable parameters and differences between NLP and vision tasks in terms of data structures and pre- training mechanisms while implementing various PETL techniques, especially for the under-explored prefix-tuning technique. Based standing of differences between NLP and video data, we propose a new variation of prefix-tuning module called parallel attention (PATT) for video-based down- stream tasks. An extensive empirical analysis on two video datasets via different frozen backhones has been carried and the findings show that the proposed PATT can effectively contribute to other PETL techniques. An effective scheme Swin- BAPAT derived from the proposed V-PETL framework achieves significantly bet- ter performance than the state-of-tt ofate-of-the-art AdaptFormer-Swi with slightly more parameters and outperforms full- full-tuning with far less parameters.
comprehensive under-
INTRODUCTION
Many vision tasks rely on fine-tuning pre-trained models to achieve good performance. One stan- dard modus operandi of transfer learning consists of two steps: pre-train model on source domain and fine-tune the entire model on a target domain (Zhuang et al. 2020). Despite that prior works have achieved promising performance, such vanilla practice of fune-tuning faced with challenges for adopting large models to downstream tasks. This full-tuning strategy requires one to update and store separate model parameters for different downstream tasks, which can be expensive and infea- sible for the era of increasingly large models from EfficientNet-based (Pham et al.) 202T) (480M parameters) Transformer-busee els, making good use of shared parameter weights deployed on the cloud can be beneficial for edge devices such as autonomous vehicles, drones who are intensive in computing and battery resources (Yuan et al. 2022). Second, the full fine-tuning strategy relies on high-quality downstream data and can hardly adapt to unseen scenarios that have large distribution shift (Kumar et al., (021), which is unlike the learning process of humans who can learn from few samples and generalize well to new circumstances. This issue has been researched directions such as zero-shot learning, few-shot learning, and continual learning (Li et al. 202 12), Another popular strategy is fine-tuning the down- stream task head, i.c., the last fully connected (FC) layer, to avoid tuning the whole backbone model, which usually leads poor performance when the target domain is large in data scale (see Figure
(Yu al. 2022) (2, 100M parameters) ones. For such large mod-

---

Under review as a conference paper at ICLR 2023
1.
Given the paradigm of fine-tuning increasingly large models, how to transfer such large models
with caannettr-accurayy trade-off is a hot topic in various domains (Gusak et al. 2022 Sung et al,
2022
Lin et al., 2020; Houlsby et al. 2019).
Taking the video-based action recognition task as an example, it can be inconvenient for deploying such large models to edge devices such
autonomous driving (Liu et al., 2019) and unmanned aerial vehicle (Li et al., 2021b) as they can heavily rely on the interaction with cloud services for adapting new environments via active learning [Wang al. [021] or continual learning (Li al.) 202 Re-training large models on the cloud are usually not cost-effective due to the expensive overheads of storage and computational resources. Furthermore, these resources are limited on edge devices such as autonomous vehicles and unmanned aerial vehicles, making the sense for developing effective fine-tuning methods with proper parameter-accuracy trade-off that can be fine-tuned on edge devices and interacting with the large models deployed on the cloud.
There have been some pioneering works for the PETL of visual models such as AdaptFormer (Chen et al., )022) and visual prompt tuning
5
Relative Accuracy Ganed
(VPT) (Jia et al.) 2022). AdaptFormer is pri-
W.tt turing (%)
Pull-tuing
marily proposed based on vision transformer (Zhai al. 2022), representing one of the state- of-the-art large models for image-based tasks. The proposed adapter module directly brings
-10
AdiplFormenSWinn
Time layyy
-15
Ours (ddore 32)
-20
Outs (dbaus 64)
from Houlsby et al. (2019) due to its conve-
Ouns (dhoott 128)
nience of being inserted any models. Im-
-25
Ouss (dhortte 256)
plementing with a large batch size of 1, 024
with 64 GPUs. Adaptformer shows promis-
10-3
10-2
10-1
10
10a
Turaible Parameters (%)
ing parameter-accuraal trade-off on video data. However, such powerful computing resource is not realistic for the usage of edge devices. Meanwhile, whether the good trade-off can be maintained for small batch size remains under-explored. Inspired by the Prompting in NLP (Liu al., (021), VPT proposes visual- prompt to fine-tune visual models for image- based tasks. According to the empirical re- sults in Chen et al. (2022), adapter modules achieves superior performance over VPT in the regimes of both self-supervised and supervised pre-training. Another concern of VPT is its modification to the original model parameters might affect the knowledge representation of backbone models. Hence, we do not continue to compare our method with VPT but comparing with the adapter on video-based downstream tasks.
Figure 1: Parameter-accuracy trade-off. Adapting backbone Swin-B (Liu al.) 2022 pre-trained on Kinetics 400 via different fine-tuning methods on the something-somethhin v2 (Goyal et al., (017) dataset. Our methods perform significantly better than the state-of -hrtthe-rrtttt adaptFormer-Swiie (Chen) et al. (2022) (our implementation with batch size 16) with slightly more tunable parameters, and outperform full-tuning with increasing margins when using larger values of diottle-
Taking the recent inspiration of the mix-and-mateh adapter (MAM adapter) (He et al. 2022a in NLP, we aim to propose a unified model for the vision domain, especially for video-based down- stream tasks. He et al. (2022a) analyzed the unified view among PETL techniques such prefix- tuning, low-rank (LoRA) adaptation, and adapter. pointing out the similarity between prefix-tuning and adapter in terms of calculating the attention. The difference is that the former performs weighted addition while the latter ones is unweighted. Note that prefix-tuning has not ever been applied to visual tasks in the form of pure visual models due to the intrinsic differences regarding pre-training methods of NLP and vision models. Another obstacle of directly applying prefix-tuning to visual tasks is the structural difference between text and vision data (we further discuss this in Section 2.3 Considering the video-based action recognition task, we propose a new variation of the prefix- tuning module called parallel attention (PATT) to adapt video-based pre-trained large models to downstream domains with varied data scales. The differences of our method comparing the original prefix-tuning NL.P are twofold: prefix calculation and the manner of insertion (see Figure 2 bb] and Figure 33. Regarding the backbone model, we focus on Video Swin Transformer (Liu et al.) 2022), one of the state-of-the-art vision models that bring competitive performance on large-scale action recognition datasets such as Kinetics 400 and 600 Kay et al. (2017).
Our main contributions can be threefold as follows:
1. We analyze different PETL techniques using the backbone model Swin Video Transformer for
2

---

Under review as a conference paper at ICLR 2023
video-based tasks, providing unified view via our V-PETL framework and investigating the impor- tance of the fine-tuning position.
2. Based the comprehensive understanding of intrinsic differences between NLP and video data regarding data structures and pre-training mechanisms, we leverage prefix-tuning to our V-PETI. with new variation called PATT.
3. Upon extensive ablation experiments regarding various effect factors, we empirically validate the promising parameter-accuracy trade-off achieved by our adjustable and easy-to-use PATT module, contributing to the existing literature of PETL techniques.
2
UNIFIED FRAMEWORK
2.1 RECAP OF VIDEO SWIN TRANSFORMER
Video Swin Transformer (Liu et al. 2022 is formed with Transformer layers (a.k.a. stages) that are consisted with 3D Video Swin Transformer blocks. With varied layers, blocks, and channel sizes, the model can be formed Swin-T, Swin-S, Swin-B, and Swin-L. The basic architecture of a 3D Swin Transformer block is shown in Figure 2 which is mainly composed of a 3D shifted window-based multi-head self-attention (3DSW-MSA) module and fully connected feed-forwant network (FFN) implemented with a 2-layer MIP. Layer normalization (LN) and residual connection are respectively performed before and after both FFN and 3DSW-MSA modules. One such Video Swin Transformer block can be represented as:
ž' = 3DSW MMSANLZZ -1) + Z
z' = FFN(LN(()) )) Z''
where z' and Z' respectively indicate the output of 3DSW-MSA and FNN modules.
Given a video input sized f x w h x 3, containing video frames with their heights and widths being h and w, respectively. The 3D patch for video data sized 2 4 x 4 x is treated as a token. Then we will have 2 x w. x h 3D tokens after a 3D patch partitioning layer. Given the 3D tokens sized 4 x 4 x h and a 3D window with the size of p m m, the self-attention module, using the regular 4
window partition strategy, will partition the 3D tokens to 1 X _ X
2p Ima im non-overlapping windows.
For shifted 3D window, the partition is shifted along the temporal, height, and width dimensions by 2 x 2 x m 2. For example, if we have an input video sized 8 x 224 x 224 x 3 and a 8 x 7 x 7 3D window, after the patch embedding, we will have 56 x 56 3D tokens with each of them sized 2 x 4 x 4 x 3. Without shifting, the non-overlapping window size will be 1 x 8 x 8 64.Then through the 3D window shifted by (4, 3) the number of 3D windows becomes 1 x 9 81.
m
The 3DSW-MSA module is formed with 3D relative position bias B E R xm xm can be represented as:
each of which
QK" vd BJV.
Attention Q. V) = SoftMar
(2)
where Q, K, V E IRxXxxxx are the query, key, and value matrices, x mxm is the number of
tokens and d is the dimension of the tokens. MSA simultaneously performs the attention mechanism for rhad heads, where the ith head can be parameterized by ((()), k
( E Rdx3d projecting the input z queries, keys, and values. Given matrix C R performing attention. the 3DSW-MSA can be calculated as:
p mxm, for
3DSW-MSA(Z -1 C) Concal(headd,, head,.) W.:
(3)
head, Attention Z ( CW()), cW."),
where W. is the parameters of linear project layer. The FNN module is composed of two linear layers with a GELU activation function in between, which can be computed as:
(' FFN Z GELU W1 + b, + b2.
(4)
where W, E Edaideen xe W₂ E Rdxdnadeen b, E Rddidden, and b2 E Rd. The value of dridden usually takes large value (e.g., dhiddon 4d).
3

---

Under review as a conference paper at ICLR 2023
2.2
RECAP OF PETL TECHNIQUES
7.x
UF
1.x
ML.P
ReLU
LayerNorm
(a) Adapter
LayerNorm
3DSW-MSA
PA
P
3DSW-MSA
LayerNorm
SoftMax
UP
LayerNorm
P
Video Swin-Transe Transformer
Proipp
Trainable
(c) Prompt- tumppt
Frozen
Prefix-tuning
Figure 2: V-PETL.: A unified view of visual PETI. techniques. They bring trainable parameters to different positions the backbone model with various manners. AdaptFormer and Prefix- Prefix-tuning respectively perform at the MLP and 3DSW-MSA modules that can adjust the number of trainable parameters via the bottleneck size of down and up projections. While prompt-tuning performed at the layer-level can adjust the length of prompts to control the tuned parameters.
Prefix-tuning (Li & Liang, 2021): The prefix-tuning approach prepends learnable prefix tokens to prefix
the keys and values of the MSA module of the model (see Figure bb). bl)). Specifically, matrices Pk, P. Bilokken xd that randomly initialized with deoken tokens and transformed from two linear layers (with parameters w
po E RRdmesate xd, and a Tanh layer in original key and value, leading the calculation of head, in Eq. to:
pk E Rdddmtaade and Wi()
between are concatenated
(5) where the concal the concatenation performed along the token dimension mimic the prefix- tuning in NLP tasks. Here, a question regarding whether this direct implementation will work for the vision domain is raised (results are Table This direct implementation empirically invalid and we make further modification on it in Section2.3
head; = Attention( Z ((()) concat( (P(() CW. (i) comcat( ( ,CW")),
Adapter (Chen al., 2022): Inspired by the works of Houlssy al. (2019): He al. (2022a) for PETL in NLP tasks, adapter (Chen et al.) (022) has been directly used for vision tasks, showing promising performance using far less tunable parameters. The number of parameters adapter is controlled by a parameter dhottle (dhotste < d), adjusting the space size of a low-dimensional representation. The adapter module first uses down-projection with Wdown dx project the feature to the lower-dimensional representation, followed by a ReLU activation function, and a up-projection with Wup E IR iddoottte xd
Ž' = ReLU Z Wdown down Wap
(6)
then two positions implementing adapter (parallel and sequential) can be respectively computed
z = FFN((N((2))) (2)) + 2 sZ',
and s ReLU(FFN(LN( 2) )) W - 2
where s a scalar, controlling the effect of the adapter (will be ablated in experiments). According
Chen al. (2022). the parallel implementation (see Figure 2 al) empirically performs better.
Prompt-tuning (Jia et al.] 2022): Prompt-tuning (see Figure [c) is inspired by the success of
prompt-tuning that adapts large scale models varied downstream NLP tasks. The idea of VPT Jia et al. 2022) is to fine-tune learnalle matrix p1-1 prompt RRddpromp xd. -dprompt < droken 1 for
4

---

Under review as a conference paper at ICLR 2023
the /th Transformer layer all Transformer layers, which are known shallow prompt and deep prompt, respectively.
z 3DSW-MSA(LN((2" p prompe: z11 + Z
where x/-1 E Rd denotes the [CLS]'s embedding for the 7th layer's input space, P Pprompt is im-
plemented by overlapping the top dpromps tokens of z (Jia al. 2022). While has also been implemented in front of the _/11 (Chen al. 2022).
Others: Other PETL techniques include ST- ST-Adapter Pan et al. (2022), LoRA (Hu et al. 2022). and BitFit (Zaken et al., 2022). ST-Adapter mainly adapts image-texi models pre- trained on large scale datasets such as 400M image-text pair proposed by CLIP (Radford et al., 2021) and the IG-3.6B used by SWAG (Singh et al. 2022) to video understanding downstream tasks, which matches and even outperforms full-tuning. LoRA approximates the optimization process by injecting learnable low-rank matrices into the attention module. This method does not show superior performance for NLP tasks in terms of parameter efficiency. Hence, we do not prioritize this direction in this work. BitFit only tunes the bias terms of the backbone models, making it very parameter-effiient,
2.3 REVISITING PREFIX-TUNE TUNNIG FOR VISUAL TASKS
The prefix implementation in Li & Liang (2021); He et al. (2022a) can be regarded as prepend- ing contextual information for downstream tasks, which similar with the pre-training process aiming to predict masked words in the process of inner loop (Brown et al., 2020). Considering the pre-training process of pure vision models, such direct implementation might not make sense for visual tasks. Although such autoregressive pre-training has been conducted in visual domain (He
et al. 2022b Tong et al., 2022), but adding prefix for a sentence input in NLP can be structurally dif- ferent with the visual domain. Specifically, masked pixels in image video data cannot be regarded as some word level semantic information (c.g., a subject or an action) as in the NLP.
Recall that the embedding state of prefix-tuning ran- domly initiated, which is known as learnable prefix but can bring random noise that later turns out affecting the convergence of the fine-tuning downstream tasks. Hence, inspired by the connection between adapter and prefix (He et al. (022a),, we avoid such learnable prefix design with random initialization and propose a parallel atten- tion (PATT) to the original attention module (see Figure 3). The adapter structure can effective control the num- ber of trainable parameters via diottle, which is similar with the effect of the middle dimension dimiddle of W pk and (ii) for preparing the prefix. Specifically, for the pu 1th layer, we use output of its previous layer Z and project it to pair of matrices K,, Vn E B via a sim- ilar mechanism of Eq. 6
SoftMax
UP
Tanh
Down
QKV
Z +-1
Figure 3: Structure of PATT. Red parts are trainable parameters calculated by the same input for preparing query, key, and value (i.e., the output of the previ- ous layer passing through layer nor- malization layer Z z
K p. Vp Tanh Z W Wdmon W up
(9)
where Tanh is the activation function used for preparing the prefix, which can be replaced by other activation func- tions such as RELU and GELU Here, we follow the original prefix implementation as its value ranges from - to 1. Given Kp and Vp. Eq. 5 can be rewritten as:
head, = Attentiinnn / (w S,,, + CW((") svv, + +VW
(10)
where is scalar for adjusting the effect of PATT. Note that without considering the physical meaning of such design, for PETL purpose, one can perform similar practise for any combinations of Q. K, and V. This brings connection to the LoRA (Hu al. 2022) method, which add parallel trainable parameters to and V. Empirically, where to perform the PATT makes little difference, but the amount of trainable parameters brings larger effect for large scale downstream domains.
2.4 V-PETL: UNIFIED VIEW ON VISUAL PETL
5

---

Under review as a conference paper at ICLR 2023
Given the PETL techniques at hand, there can be many potential combinations leading to good parameter- accuracy trade-off. However, it is unrealistic to exhaus- tively test all the methods for specific downstream task. Other than probing such solution via evolutionary search as in Zhang et al. (2022). we aim to propose more under- standable models by empirically analyzing the effect of different designs independently. According to the prelim- inary results shwon in [i we argue that the position and amount of parameters are important for PETL tech- niques, especially when the target domain is not small.
Table I: Comparison of independently
fine-tuning varied positions of the video
swin transformer block on SSv2.
Position
# Params
Top-1 (%)
Full-tuning Tune FC Layer
87 .28
50.99
0.18M
24.13
LayerNorm
0.02M
14.35
Attn, Proj
6.99M
47.58
Attn, QKV
20.98M
50.02
Attn. SuftMax
0.95M
27.67
LayerNorm
0.02M
14.62
To verify the importance of position and tuned parameter amount, we independently tune different modules of the backbone model. Table [ shows the results. We can see that the attention module's QKV layer has 20.98M pa- rameters while the MLP module has the most number of parameters of 55.90M. Tuning positions with more parameters, will lead to better performance for SSv2. Thanks to the bottleneck mecha- nism of adapter and prefix-tuning, one can effectively achieve good parameter-accunall trade-off. As such, we derive a model called work by using the parallel adapter and our PATT leverage the adaption of pre-trained backbone model at the positions of attention and MLP modules, respectively. In addition to adapter and PATT, we also fine-tune the last fully connected layer as has relatively smaller amount of tunable param- eters (i.e, 0.18M) than adapter and PATT.
MLP, FC1
27.97M
47.10
MLP. FC2
27.93M
45.32
DownSample 2.76M
27.53
(Swin-BAPAT) from the V-PETI. frame-
3 EXPERIMENTS
3.1 EXPERIMENTAL SETTINGS
Video Datasets: Something-sometthing v2 (SSv2 (Goyal et al., 2017)) It has 108,499 short videos for 174 human-object interaction categories with durations between 2 to 6 seconds. The challenge of this dataset is that it contains 23, 137 distinct object names with an imbalanced distribution. The original dataset split into train, validation, and test sets with a ratio of 1:1:1. The extended version (SSv2) of this dataset is consisted of 168, 913 training samples, 24, 777 validation samples, and 27, 157 testing samples with the sample number of action labels. The training and testing samples are used. HMDB51 (Kuchne et al.] 2011) contains 6, 766 video samples for 51 action categories including videos of varied visible body parts, camera motion, camera view, and clip quality. All video samples have at least 101 clips and minimum height of 60 pixels for actors. The original dataset has three splits of training and evaluation. We follow existing work Chen el al. (022) by using the first training and evaluation split that has 3, 570 and 1, 530 samples, respectively. Image Datasets: Following the experimental set ups in AdaptFormer. three datasets CIFAIR- Krizhevsky et al. (2009). Street View House Numbers (SVHN) Goodfellow et al. (2013). and Food- 101 Bossard et al. (2014) are used. CIFAIR- 100 has 50, 000 and 10. 000 training and validation images, respectively. with the resolution of 32 x 32 and 100 categories: SVHN is a digit classification dataset that has 73, 257 training sample and 26, 032 testing samples; Food-101 includes 101k images of 101 food categories with each of them has 750 training and 250 testing samples,
Implementation details: It is worth noting that big batch size (i.e., 1, 024) and the number of input video frames (i.e., 32 frames) can greatly benefit good performance (Carreira & Zisserman, 2017] Liu et al. 2022) Chen et al.] 2022), which usually requires GPU clusters to enable the training. AdaptFormer (Chen et al. (022)) uses such powerful GPU cluster to achieve good performance. However, good performance might not hold when the batch size is small. Following the more common hardware device setup, we use 4 GeForce 3090 GPUs for all experiments, leading to a batch size of 64. All the experiments are fine-tuned for 70 epochs. We use the Swin-BfT model pre-trained on Kinetics 400 and 600. For HMDB51, we report the results without tuning the FC layer due the significant effect of the FC layer on relatively small scale dataset. Following Chen et al. 2022), we do not perform regularization strategies such as mixup, cutmix, color jittering,

---

Under review as a conference paper at ICLR 2023
Table 2: Comparison of Top-1 accuracy using varied amount of parameters adjusted by dbottles different pre-training domains, and the number of frames with other fine-tuning strategies.
SSv2
HMDB51
Method
diottle Pre-training
# Prames
# Params Top-1 (%) Params Top-1 1 (%)
Full-tuning
Kinetics 400
8
87.82M
50.99
87.69M
68.07
Tune FC Layer
Kinetics 400
8
0.18M
24.13
0.05M
71.28
BiiFi (Zaken al. 2022
Kinetics 400
8
1.29M
45.94
1.11M
68.26
AdaptFormer-Swii (Chen et al.
2022
64
Kinetics 400
8
1.73M
40.80
1.61M
68.66
Prefix- tiiii-tttnng (Li & Liang 2021
128
Kinetics 400
8
6.57M
39.46
6.40M
56.13
Our Swin- BAPAT (wlo Adapter)
32
Kinetics 400
8
1.35M
46.26
1.77M
69.51
Our Swin- BAPAT (wlo Adapter)
64
Kinetics 400
8
M.51M
49.23
2.34M
71.34
Our Swin-BAPAT (w/o Adapter)
128
Kinetics 400
8
4.83M
52.57
4.65M
70.56
Our Swin- BAPAT (wlo Adapter)
256
Kinetics 400
9.45M
52.71
9.27M
70.23
Our Swin- BAPAT
32
Kinetics 400
8
2.91M
49.63
2.74M
68.20
Our Swin- BAPAT
64
Kineties 400
8
4.07M
51.80
3,89M
70.10
Swin-BAPAY
128
Kinetics 400
6.38M
$3.36
6.20M
71.93
Our Swin- BAPAT
256
Kinetics 400
11.00M 8
53.98
10.83M
69.64
Our Swin- BAPAT
256
Kinetics 400
8
11.00M
53.98
10.83M
69.64
Swin- BAPAT
256
Kinetics 600
8
11.00M
54.06
10.83M
69.90
Our Swin- BAPAT
256
ImageNet-22K
8
11.00M
43.56
10.83M
59.89
Our Swin- BAPAT
128
Kinetics 400
8
6.38M
53.36
6.20M
71.93
Our Swin- BAPAT
128
Kinetics 400
16
6.38M
63.14
6.20M
75.67
etc. Our PATT module is convenient be applied to other Transformer- based models. Hence, we respectively adopt ViT-B models from MAE (He et al. (022b) and VideoMAF (Tong et al. 2022) to conduct further comparison on video and image datasets, which follows the self-supervised pre- training setting" in Chen et al. (2022) except that the batch size is set to 256 instead of 1, 024.
Baselines:
We mainly compare our method SWITTBAPAT with three baselines as follows:
(I) Full-tuning: set all the parameters learnable and tune the whole model initiated with the pre- trained weights. (2) Tune FC layer: tune the last fully connected layer and freeze pre-trained pa-
rameters of the whole backbone model. (3) AdaptFormer-Swin method introduced by CCen et al.
(022) that adds a parallel adapter to the MLP module in each block of the backbone model. (4) Prefix-tuning: the direct implementation of prefix-tuning used in NLP defined in Eq. 5 (5) BitFit:
by tuning the bias of the backbone model together with the FC layer.
3.2 THE EFFECT OF DIFFERENT PETL TECHNIQUES
Table 2 shows the results of different PETL techniques. From the results of four baseline methods, full-tuning performs the best for the large-scale dataset SSv2, whereas tuning the FC layer achieves superior performance over other PETL techniques on HMDB51. This is due to the fact that down- stream tasks with relatively larger scale datasets are more parameter hungry for good convergence. On the contrary, small datasets can make good use of the knowledge from the source domain with slight effort of adaption via an FC layer. Here, question regarding the effect of this FC layer when using it together with other PETL techniques has not been investigated. As this FC layer having small amount of tunable parameters can already make a big difference, performing better than full- tuning and other PETL techniques and rendering them not effective for small-scale datasets. As such, we further examine this question in Section A.
We test different amount of parameters adjusted by Sbottlee taking its values to 32, 64, 128 and 256. The second and third groups (without or with Adapter, respectively) of results in Table shows that larger values of Sbottle can benefit the fine-tuning with slightly more overhead of parameters on large-scale datasets such as SSv2. All results of our Swin-BAPAY outperform the state-of- the-art AdaptFormer-Swin with big margin (using the smallest value Shottle = 32 can improve AdaptFormer-Swinn by almost 25%). While without using Adapter, our method still outperforms baselines AdaptFormer-Swee Swinn and BitFit with roughly similar amount of parameters. When Sbottle larger than 64, our Swin-BAPAA starts to perform better than full-tuning on both datasets with proper parameter-accuragg trade-off, validating the effectiveness of our Swin-BAPAT for PETL.

---

Under review as a conference paper at ICLR 2023
70
00
Table 3: Top-1 accuracy (%) using differ- ent scalar values on two datasets: SSv2 and HMDB51. The dhottle is set 128; pre- training is based on Kinetics 400.
Top- Accuraty (%)
10
30-
78 SLB B256 X400
18 508 B236 10000
Scalar
SSv2
HMDB51
20
10 SLB B32 K400
R8 505
D64 K-000
X400
full-tuning
50,99
71.28
10
Swin
Tun
Tune FC Layer
24.13
68.07
AdaptFormer-Swi
40.80
68.66
Numbrral Epoch
0.2
47.46
69.38
0.5
52.84
71.87
Figure 4: Top-1 accuracy of different settings on
0.8
53.36
71.93
SSv2 throughout training process. F: frame, S:
1.0
53.29
70.89
scalar, B: doottles K: pre-training domain.
3.3 THE EFFECT OF DIFFERENT TRAINING DOMAINS
The knowledge from the pre-trained model is learned from the source domain. We test two different models tre-trained on large-scale datasets: Kinetics 400, Kinetics 600, and ImageNet-222 22K.. Findings show that both two models pre- pre-trained such large-scale datasets can benefit our proposed PETL strategy with the latter being slightly more significant (see the third group of comparison in Table 2). This is due to the fact that Kinectics 600 is larger than its 400 version and brings more knowledge to the pre-trained model, benefiting more downstream tasks. However, image-based pre-training cannot perform good as video-based pre-training due the larger domain gap.
3.4 THE EFFECT OF DIFFERENT VIDEO INPUT SIZE
We also test whether our method is robust to increased number of input video frames. It is worth noting that larger number of input video frames usually can bring more spatial temporal information, benefiting data-driven models to learn more distinguishable features while keeping the model size remaining the same. The last group of comparisons in Table 2ihows that using double- sized video input (i.e., 16 frames) large-scale datasets. The improvements (increased 9.78% from 53.36% to 63. 14% on SSv2, and 3. 3.74% from 71.93% to 75.67% on HMDB51) 1) more significant than other factors such as diottle and pre-training domain (around 1% to 2%). The top line in Figure 4 visualizes the significant effect of increasing the number of input video frames. These results suggest that our Swin- Swin-BAPAT BAPAT can be promising for increased frames of video input.
greatly improve the performance of action recognition on both small and
3.5 THE EFFECT OF DIFFERENT SCALE OF PATT
Recall that the effect of our PATT on pre- trained models can be adjusted by the variable 8 in Eq. 10 Table 3 shows that adopting the value of 0.8 deliver consistent best perfor- mances on both datasets SSv2 and HMDB51 under our experimental setting. Smaller val- ues of & will quantitatively reduce the effect of our PATT module on the knowledge trans- fer while large values will increase the effect of our PATT module. The good performance achieved via taking an effective scale of 0.8 in- dicates that our PATT module plays an impor- tant role in the knowledge transfer. However, even larger values over 0.8 can affect the im- portance of original knowledge thereof the pre- trained model. Hence, proper valued scalar s is essential for balancing the role of PATT and
Table 4: Ablation of different implementation po- sitions of PATT defined in Eq. 10 e.g., Ours (K, V) indicates inserting PATT to the query and key of 3DSW-MSA modules. Pre-training 600. doottie is set to 128; Scalar 8 is set to 0.8.
Kinetics
SSv2
HMDB51
Method
Params
Top-1
# Params
Top-1
Full-tuning
87.82M
50.99
87.69M
68.07
Concat (K. V)
6.38M
15.61
6.20M
20.98
No 2- (K.
8. 8.74M
51.06
8.56M
67.41
Ours (Q, K)
6.38M
45.49
6.20M
68.92
Ours (K, V)
6.38M
53.38
6.20M
71.41
Ours (Q.
6.38M
53,24
6.20M
71.74
Ours Q K. V)
7.93M
53,23
7,63M
69.57
8

---

Under review as a conference paper at ICLR 2023
pre-trained backbone model. Note this can be learnable parameter upon specific implementation, here we empirically verified the effect of the scalar.
3.6 THE EFFECT OF DIFFERENT METHODS YIELD FROM V-PETL
We have argued that, especially for relative large downstream datasets, the position and the amount of trainable parameters are important for parameter-efficien efficient transfer learning in Section 2.4 The proposed Swin-BAPAT BAPAT is one of instantiated models from the V-PETL framework regarding the in-
sert position of our PATT. Other instantiations can be inserted into different positions such as query, key, and value of the attention module. We further instantiate other variations of our Swin-BAPAT by inserting PATT to different positions. Table shows the results. Findings show that inserting to the value position of 3DSW-MSA can contribute more than inserting other two positions. While inserting to query of key makes little difference for the performance. This is due to the fact that query and key make the calculation of the attention mask. Hence, inserting either one of them will lead to a similar effect. On one hand, these results, to some extent, justify the original design of preiixtunnin that bring learnable prefix to key and value of the attention module. On the other hand, it indicates that our claim regarding the unified view of PETL for visual tasks is reasonable. In Table we also ablate the designs of PATT regarding concatenating Kp and Vp (i.e., Coneat IK, VJ), and using trainable parameters to generate Kp and vp (i.c., No Z IK, VD.
3.7
COMPARISON ON VARIED TASKS VIA SUPERVIDD PRE-TRAINEL MODELS
Table 5 shows the comparison with AdaptFormer-64 (Chen et al., 2022) and VPT (Jia et al. 2022 on both image- and video-based downstream tasks. Our method VIT-BAPAT still shows promising parameter-accuracy trade-off via much smaller batch size, which is more convenient for reproduction on the general single server with GPUs The underperformance on SSv2 (better than full-tuning) can be due to the smaller batch size as SSv2 is much larger than other compared datasets and can be more relying on larger batch size. In real-world application scenarios, small dataset can be the more common case, which confirms our contributions.
Table 5: Comparison of Top-1 accuracy via ViT-B models from MAE and VideoMAE pre-trained with self-supervised learning for image and video datasets, respectively,
Avg.
Image SVHN
Video
Method
Params (M)
CIFAR-100 100
Food-101
SSv2
HMDB51
Full-tuning
86.04 (100%)
85,90
97.67
90.09
53.97
46.41
Tunc FC Layer
0.07 (0.08%)
69.83 (-16.07)
66.91 (-30.76)
69.74 (-20.35)
29.23 (-24.74)
49.84 (+3.43)
VPT (Jia et 2022
0.08 (0.09%%
82.44 (-3.46)
94.02 (-3.65)
82.98 (-7.11)
43.73 (-10.24)
52.67 (+6.26)
AdaptFormer-64
1.26 (1.46%%
85.90 (0.00)
96.89 (-0.78)
87.61 (-2.48)
59.02 (+5.05)
55.69 (+9.28)
Our ViT-BAPAT-32
2.13 (2.47%)
86.29 (+0.39)
97.18 (-0.49)
87.37 (-2.72)
57.78 (+3.81)
57.18 (+10.77)
Our ViT-BAPAT-64
3.02 (3.51%)
86.35 (+0.45)
97.18 (-0.49)
87.53 (-2.56)
57.55 (43.58)
57.18 (+10.77)
Our ViT-BAPAT- 128
4.79 (5.56%)
86.47 (+0.57)
97.28 (-0.39)
87.66 (-2.43)
56.97 (+3.00)
57.70 (+11.29)
Our VTT-BAPXT-256
8.33 (9.68%)
86.55 (+0.65)
97.24 (-0.43)
87,68 (-2.41)
56.53 (+2.56)
57.31 (+10.90)
4 CONCLUSION
In this paper, we introduced a V-PETL framework for exploiting good parameter-accuracy trade- off around adapting video-based pre-trained large models downstream tasks. Our Swin-BAPAT method derived from the V-PETL with a variation of prefix-tuning known as PATT can effectively bring good parameter-accuracy trade-off on downstream tasks. The proposed PATT can be eas- ily plugged to the attention module of other transformer-lis models. Mcanwhile, the amount of trainable parameter can be easily adjusted by the parameter dhottle- With small amount over- head on trainable parameters, our method performs significantly better than state-of-the- method AdapFormer-Swin and full-tuning on the datasets SSv2 and HMDB51 via small batch size, vali- dating our contribution to the literature of PETL. In the future we will test our proposed model on more action recognition datasets surveyed in Sun et al. (2022) under more learning regimes such as zero/few-shh learning, active learning and continual learning with other pre- mre-trainin methods such as visual-langee language models. We will also explore other backbone models, activation functions for PATT, and PETI. techniques such LoRA for visual tasks.
9

---

Under review as a conference paper at ICLR 2023
REFERENCES
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mining discriminative compo- nents with random forests. In European conference on computer vision, pp. 446-461. Springer, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2299-6308, 2017.
Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang, Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. arXiv preprint 1:Xiv:2205.13535, 2022.
lan J Goodfellow, Yaroslav Bulatov, Julian Ibarz, Sacha Arnoud, and Vinay Shet. Multi-digit number recognition from street view imagery using deep convolutional neural networks. arXiv preprint arXiv:1312.6082, 2013.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Julia Gusak, Daria Cherniuk, Alena Shilova, Alexandr Katrutsa, Daniel Bershatsky, Xunyi Zhao, Lionel Eyraud-Dubois, Oleh Shliazhko, Denis Dimitrov, Ivan Oseledets, and Olivier Beaumont. Survey on efficient training of large neural networks. In Lud De Raedt (ed.), Proceedings of the Thirty-First International Joint Conference Artificial Intelligence, IJCAI-22, pp. 5494 5501. International Joint Conferences on Artificial Intelligence Organization, 7 2022. doi: 10.24963/ jjcai.2022/766 htttss:s:://oo org 24963/156aaaa 2aliit.022///2 7669 Survey Track.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor unified view of parameter-efffcieet transfer learning. In International Conference on Learning Representations, 2022a.
and Graham Neubig. Towards
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li. Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022b.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo. Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nip. In International Conference on Machine Learning, pp. 2790-2799. PMIR, 2019.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference on Learning Representations, 2022. URL htttps://openrevieeett net forum?
id=nZeVKeeFFFF
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning, arXiv preprint arXiv:2203.12119 2022.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:/705.0690 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Hildegard Kuehne, Hueihan Jhuang, Estibaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: large video database for human motion recognition. In 2017 International conference on computer vision, pp. 2556-2563, IEEE, 201
10

---

Under review as a conference paper at ICLR 2023
Ananya Kumar, Aditi Raghunathan, Robbie Matthew Jones, Tengyu Ma, and Perey Liang. Fine- tuning can distort pretrained features and underperform cutt-f-distribution. In International Con- ference Learning Representations, 2021.
Tianjiao Li, Qiuhong Ke, Hossein Rahmani, Rui En Ho, Henghui Ding, and Jun Liu. Else-net: Elastic semantic network for continual action recognition from skeleton data. In Proceedings of the IEEE/CVF Interaational Conference on Computer Vision, pp. 13434-13443, 2021a.
Tianjiao Li, Jun Liu, Wei Zhang, Yun Ni, Wenqian Wang, and Zhiheng Li. Uav-human: A large benchmark for human behavior understanding with unmanned aerial vehicles. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16266-16275, 2021b.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume 1: Long Pupers),
pp. 4582-4597, 2021.
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter-efficien transfer learning. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pp. 441 441-459, 2020.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- 2021.
cessing. arXiv preprint
Shaoshan Liu, Liangkai Liu, Jie Tang, Bo Yu, Yifan Wang, and Weisong Shi. Edge computing for autonomous driving: Opportunities and challenges. Proceedings of the IEEE, 107()):167711716 2019.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin trans- former. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recogni- 2022.
tion, pp.
Junting Pan, Ziyi Lin, Xiatian Zhu, Jing Shao, and Hongsheng Li. St-adapter: Parameter-efffcient image-to-video transfer learning for action recognition. arXiv preprint arXiv:2206. (tt.::2220..35599 2022.
Hieu Pham, Zihang Dai, Qizhe Xie, and Quoe V Le. Meta pseudo labels. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11557-11568, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateck Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens van der Maaten. Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Zehua Sun, Qiuhong Ke, Hossein Rahmani, Mohammed Bennamoun, Gang Wang, and Jun Liu. Human action recognition from various data modalities: A review. IEEE transactions on pattern analysis and machine intelligence, 2022.
Yi-Lin Sung. Jaemin Cho, and Mohit Bansal. Vl-adapter: Parameter-efficient transfer learning for vision-and--anguuaee tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5227-5237, 2022.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. arXiv preprint arXiv:2203. /2602, 2022.
Xiaohan Wang, Linchao Zhu, Heng Wang, and Yi Yang. Interactive prototype learning for egocen- tric action recognition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8168-8177, 2021.
11

---

Under review as a conference paper at ICLR 2023
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Sha Yuan, Hanyu Zhao, Shuai Zhao, Jiahong Leng, Yangxiao Liang, Xiaozhi Wang, Jifan Yu, Xin Lv, Zhou Shao, Jiaao He, et al. A roadmap for big model. arXiv preprini arXiv:2203. 14101, 2022.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-effcient fine-tuning for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Short Pupers), pp. 1-9, 2022.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
12104-12113, 2022.
Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu. arXiv:2206. 04673, 2022.
Neural prompt search.
arXiv preprint
Fuzhen Zhuang, Zhiyuan Qi, Keyu Duan, Dongbo Xi, Yongchun Zhu, Hengshu Zhu, Hui Xiong, and Qing He. A comprehensive survey on transfer learning. Proceedings of the IEEE, 109(1); 43-76, 2020.
A APPENDIX
A.1 THE EFFECT OF FC LAYER FOR SMALL SCALE DOWNSTREAM TASKS
Table 6: Results of with or without tuning the FC layer on the small scale dataset HMDB51.
with FC layer
without FC layer
Method
doottle
Pre-training
# Frames
Purams
Top-1 (%)
Params
Top-1 (%)
Our Swin-BAPAT
BAPAT
32
Kinetics 400
8
2.79M
65.97
2.74M
68.20
Our SWin-BAPAT
64
Kinetics 400
8
3.94M
67.28
3.89M
10
Our -
128
Kinetics 400
8
6.25M
66.75
6.20M
71.93
Our Swin-BAPAT
256
Kinetics 400
10.88M 8
67.67
10.83M
69.64
Our BAATT
Kinetics
8
10.88M
67.67
10.83M
69.64
Our Swin-BAPAY
256
Kinetics 600
8
10.88M
67.41
10.83M
69.90
Our Swin-BAPAT
128
Kinetics 400
8
6,25M
66.75
6.20M
71.93
Our Swin-BAPAT
128
Kinetics 400
16
6,25M
70.56
6.20M
75.67
Our Swin-BAPAT
128
Kinetics 400 32
6.25M
74.82
6.20M
76.46
For the small dataset HMDB51, due to the good parameter-accuracy trade-off achieved by fine- tuning the FC layer only, adding the FC layer cannot bring extra improvement our proposed method. Without sufficient taining data, full-tuning also cannot perform well (see results in Table 2). As such, small datasets do not need rely on large models but can make use large models with light transfer. Instead, without tuning the FC layer, our Swin-BAPAT can perform better than fine-tuning the FC layer with small amount of extra trainable parameters (see results in Table 6) validating the good
trade-off of our method.
12


Reviews:
[{"review_id": "o1wElns3Xi7", "reviewer": "Reviewer_oYYz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a module inserted into a pre-trained model for parameter efficient fine-tuning. The authors empirically combine adapter, adapt pre-fix tuning, and verify the effectiveness of the design through a series of experiments. The proposed method achieves very competitive results and has the potential to outperform full fine-tuning.", "summary_of_the_review": "At first glance, the proposed method seems to be a simple yet effective design that combines and adapts previous techniques of parameter efficient fine-tuning. However, after close examination, some critical experiments and ablations are not provided to support this paper.", "strength_and_weaknesses": "## Strength\n\n1. This paper proposed a design for parameter efficient fine-tuning and demonstrate its effectiveness in a video setting.\n\n1. The authors conducted some analyses to (partially) validate the design choices of the proposed method.\n\n## Weakness\n\n1. This is mainly an empirical paper. Therefore, the reviewer would expect more extensive and rigorous experiments to demonstrate the generalizability and effectiveness of the proposed method. For example, different architectures, and other combinations of pre-training and fine-tuning datasets.\n\n1. Some conditions are missing in Table 1:\n    - Section 2.2 and Figure 2 mentioned three types of most common baselines, but the prompt tuning baseline is missing\n    - In CV or NLP, typically more trainable parameters lead to better performance. Therefore, the authors should compare their method with baselines under similar trainable parameters.\n    - Since the method is the combination of adapter, PATT (a variant of prefix-tuning), and fc layer fine-tuning, the author should show the results of different combinations of baselines.\n\n1. Some critical ablations of the proposed method are missing:\n    - Since the final method is the combination of adapter, PATT, and fc layer fine-tuning, the authors should ablate this.\n    - The proposed PATT is a variant of prefix-tuning. Thus the author should also ablate the design of PATT. Specifically, (1) with or without the weight $s$, (2) weighted addition v.s. concatenation of $K_p$ and $V_p$, and (3) $Z^{l-1}$ vs some trainable parameters as the input to Equation 9 for generating $K_p$ and $V_p$.\n\n1. It would also be good to analyze why the proposed design is better (Eg. mathematically, from the perspective of gradient propagation, or anything else). Otherwise, it looks like the authors simply enumerate some possible/common design choices and do a manual architecture search.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "hN6R-ktrHd", "reviewer": "Reviewer_GJuJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new framework to adapt large video-based models to down-stream tasks with a parameter-accuracy trade-off. It analyzes different PETL techniques and investigates the importance of fine-tuning position of their methods. In order to better transfer prefix-tuning from NLP to vision task, it compares differences between NLP and video data regarding data structures and pre-training mechanisms.  ", "summary_of_the_review": "The proposed method seems to achieve the best performance under the condition of limited computing resources. This paper should demonstrate the superiority of the method under fairer conditions or give sufficient reason analysis.", "strength_and_weaknesses": "Strengths:\n● The paper is well written and well structured.\n● The authors considers the situation of insufficient computing resources. Under the corresponding experimental settings, the experimental results demonstrate the superiority of the method proposed in paper.  \nWeakness:\n● As one of the baselines, Adaptformer can achieve excellent performance (59.02 on SSv2 and 55.69 on HMDB51) after larger batch size training.  Why does reducing the batch size has such a big impact on performance of Adaptformer? It is uncertain whether authors have explored the hyperparameters of Adaptformer under small batch size setting. In other words, authors can increase the batch size to compare the proposed method and Adaptformer.\n● The paper only shows the experimental results but does not elaborate on the reasons for the gains brought by the method.\nMinors:\n● There are some minor mistakes in the experimental table: first two lines in Table 1 appear to have numerical errors where \"0.11M\" is inconsistent with\"0.18M\" in Table 3, and \"Tune FC layer\" and \"Full-tuning\" should be swapped.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "XfuM_Ev8mGO", "reviewer": "Reviewer_uEYB", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Motivated by recent studies on parameter-efficient learning of Transformers in NLP (He et al., 2022) and vision domain (Jia et all., 2022), the authors proposed to extend them to video models. Numerous ablation studies are conducted to find the best configuration for incorporating parameter efficient approaches in video Transformers, which turned out to be consistent with the previous observation (He et al., 2022) where the self-attention and MLP layers in Transformers are modified with a prefix-tuning-like approach and adaptor, respectively. The proposed method is evaluated in two video downstream tasks and demonstrated encouraging results.", "summary_of_the_review": "Overall, I lean towards rejecting this paper since the ideas and results from the paper largely overlap with the prior work (He et al., 2022) and the experiments only demonstrate the results in videos. ", "strength_and_weaknesses": "Strength:\n- The motivation for applying parameter-efficient learning to videos is sound and reasonable.\n- Ablation studies and empirical results are strong and convincing.\n\nWeakness:\n1. Novelty\nThe ideas presented in the paper–steering the Transformer via a prefix-tuning-like modification to MHSA and adaptor-like modification to the MLP–are already explored by He et al., 2022, which leads to a very similar model. Although there are some differences in details, such as how we incorporate prefix-tuning in self-attention, these are rather minor and can be considered as the special case of the prior work (He et al., 2022). Although I agree that extending the observations to videos can be valuable, the impact is still not significant considering that success in the NLP domain tends to easily extend to the visual domains.\n\n\n2. Experiments\n- Although the same method is applicable to images, the paper is evaluated only on videos. To understand the benefits and limitations of the proposed method better, it would be great to see the results in image downstream tasks as in the prior works (e.g., Chen et al., 2022).\n\n- The comparisons to AdaptFormer (Chen et al., 2022) in Table 2 need more careful considerations. It is not easy to directly compare the two methods (AdaptFormer and the proposed method) since the number of parameters are different (AdaptFormer uses fewer parameters). Considering the results from Chen et al., 2022 that the performance of AdaptFormer also increases proportionally to the number of parameters, it would be more desirable to roughly match the parameters and compare the performance in multiple scales. \n\n\n3. Presentations\n- There are some minor presentation issues. \n  1) the first dimension of B (in the line above Eq.(2)) should be t->p. \n  2) in Table 1, the first two rows should be switched. \n\n- It would also be interesting to consider comparisons with much more parameter-efficient alternatives such as BitFit (Ben-Zaken et al., 2022).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "LwGoy-d7cH", "reviewer": "Reviewer_EH6p", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The goal of this paper is to reduce the parameter usage when adapting pretrained models to video tasks. To this end, the authors propose a new parameter-efficient method, which adds the learnable parameters to generate additional keys and values. Unlike the prior Prefix-tuning, the authors further reduce trainable parameters by introducing an autoencoder-like modules for generating keys and values. In the experiment, the authors show the improvements when adapting a Kinetics 400 pretrained models to SSv2 and HMDB51 datasets. ", "summary_of_the_review": "While the authors propose a new parameter-efficient method for video classification tasks, there exist some concerns and questions in the experiment sections. More experimental setups and baselines should be considered for a more complete and convincing empirical results. Given the current status of the paper, I would suggest the score \"marginally below the acceptance threshold\".", "strength_and_weaknesses": "**Strengthes**\n- The paper presentation is good, and the authors clearly show how the proposed method is designed with easy-to-understand figures (Figure 2 and 3).\n- The authors presented several analyses on the hyper-parameters (Table 2, 3, 4, and 5).\n\n**Weaknesses**\n- Paper title does not properly describe the paper content: Since the a large portion of the paper deals with the video data and video recognition tasks, I would suggest the author either include more different types of vision tasks or narrow the title to \"for video recognition tasks\".\n\n\n- In section 2.3, \"prefix implementation in NLP can be regarded as providing conditional information for the downstream tasks, which is similar with the pre-training process where words are masked\". The sentence is confusing to me, since the connection between the prefix tuning and masking text pretraining does not look similar to me. I would suggest to elaborate this motivation and provide a more reasonable motivation why the prefix tuning is suitable for video tasks. \n\n- CLIP is not one of parameter-efficient methods, so it should be removed from Section 2.2\n\n- Less convincing experiments: \n\n1. Missing an important baseline - ST-Adapter [1]: Since the paper addresses the video tasks, one important baseline (ST-Adapter [1] ) should be included in the experiment. ST-Adapter present several video benchmarks, including SSv2 and K-400 datasets with different pretrained weights.\n\n2. *Comparison to AdapterFormer*: Although the authors compare AdapterFormer, there are still some concerns in the comparison. First, the experimental setup of AdapterFormer paper is different from the proposed method, since AdapterFormer uses different settings (pretrained on IN-21k and adapt to SSv2). I would suggest the authors also include their experimental setups (same backbone, same pretraining datasets, and same downstream tasks) to provide a complete comparison. It is unclear whether the baseline is properly tuned and well reproduced. Also, the authors show the number of parameters of AdapterFormer is only 1.73M, while the AdapterFormer, which is based on adapter, can certainly be improved by increasing the hidden state size of adapters. \n\n3. Experiment setups are still limited. The authors only show a single model architecture, video SwinTransformer, and I would suggest the authors extend to other model architecture. I would be interested in how the proposed method performs under ViT with ImageNet-21k perform on K400.\n\nRef:\n[1] \"ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning\", Pan et al., NeurIPS 2022", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject