Facing the Elephant in the Room: Visual Prompt Tuning or Full finetuning?
Authors: Cheng Han
Qifan Wang
Yiming Cui
Wenguan Wang
Lifu Huang
Siyuan Qi
Dongfang Liu
Venue: ICLR 2024 poster (2024)
Keywords: Visual prompt tuning
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: As the scale of vision models continues to grow, the emergence of Visual Prompt Tuning (VPT) as a parameter-efficient transfer learning technique has gained attention due to its superior performance compared to traditional full-finetuning. However, the conditions favoring VPT (the "when") and the underlying rationale (the "why") remain unclear. In this paper, we conduct a comprehensive analysis across 19 distinct datasets and tasks. To understand the "when" aspect, we identify the scenarios where VPT proves favorable by two dimensions: task objectives and data distributions. We find that VPT is preferrable when there is 1) a substantial disparity between the original and the downstream task objectives ($e.g.$, transitioning from classification to counting), or 2) a notable similarity in data distributions between the two tasks ($e.g.$, both involve natural images). In exploring the "why" dimension, our results indicate VPT's success cannot be attributed solely to overfitting and optimization considerations. The unique way VPT preserves original features and adds parameters appears to be a pivotal factor. Our study provides insights into VPT's mechanisms, and offers guidance for its optimal utilization.

Published as a conference paper at ICLR 2024
FACING THE ELEPHANT IN THE ROOM: VISUAL PROMPT TUNING OR FULL FINETUNING?
Cheng Han Qifan Wang", Yiming Cui', Wenguan Wang", Lifu Huang", Siyuan Qi", Dongfang Liu
Rochester Institute of Technology', University of Missouri Kansas City², Meta AT University of Florida", Zhejiang University", Virginia Tech BIGAI fch7858, dongfang 1iut edu, wqfer wttrriib.com. cuiyimingeuf lifuh@vt edu, wenguanwang ai ai@gmaiil mimmail.com com, sygi8bigai.ei
ABSTRACT
As the scale of vision models continues to grow, the emergence of Visual Prompt Tuning (VPT) as a parameter-efficien transfer learning technique has gained at- tention due to its superior performance compared However, the conditions favoring VPT (the "when") and the underlying rationale (the "why") remain unclear. In this paper, we conduct comprehensive analysis across 19 distinct datasets and tasks. To understand the "when" aspect, we identify the scenarios where VPT proves favorable by two dimensions: task objectives and data distributions. We find that VPT is preferrable when there is 1) substantial disparity between the original and the downstream task objectives (e.g., transition- ing from classification counting), 2) a similarity in data distributions between the two tasks (e.g., both involve natural images). In exploring the "why" dimen- sion, our results indicate VPT's success cannot be attributed solely to overfitting and optimization considerations. The unique way VPT preserves original features and adds parameters appears to be a pivotal factor. Our study provides insights into VPT's mechanisms, and offers guidance for its optimal utilization. The code is available at
traditional full-fnettningg
1/VPTTor-FT
INTRODUCTION
Recent advancements in artificial intelligence 38
especially large models in natural lan- 58 77 [31]. have led to the development of large-scale vision models
93 3
guage processing [13 [33
(e.g.. BiT [53], VIT [24). Swin (66). Florence (124) that have revolutionized various tasks. These
models are typically trained on extensive datasets (e.g. ImageNet-21k 2k [99] Open Images (55) and then finetured to adapt to specific downstream tasks [47] (e.g.. Coco-stuff 77 FGVC 14 VTAB-
1k (127).. As models drastically scale up to boost performance, full finetuning that unfreezes all parameters for the update becomes computationally and storage-intensive [11]] and impractical. This traditional method also lead to the loss of valuable knowledge acquired during pretraining.
thereby limiting the models' generalizability 49 134].
To address this issue, the search for parameter-efficient finetuning methods in vision an ongo- ing endeavor. These parameter-efficient approaches freeze most parts of the pretrained model, and only tune the rest or insert customized learnable modules, which significantly reduce the number of learnable parameters compared to full finetuning [16]
85 129]. Among all prompting 64 34), stands out one of the most prominent techniques in this field. VPT in- troduces a small number of extra trainable parameters (typically less than 1% of model parameters) in the input space of the transformer layers while keeping the backbone frozen. For the first time, it achieves superior performance over full finctuning in image recognition, and quickly adapted into
48
69
these methods, visual prompt tuning 49) which is again inspired by language-domanl
works (68] 102 65
other visual tasks such image segmentation 662 120 130 110), image captioning [135] etc.
With VPT's growing popularity B 99 112), a research question naturally arises: when and why does visual prompt tuning (VPT) outperform full finetuning (FT) transfer learning paradigm?
Corresponding author National Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelli-
gence

---

Published as a conference paper at ICLR 2024
To address this question, conducted extensive experiments on 19 diverse datasets and tasks, wherein VPT outperformed FT in instances. To discern when VPT is preferred, we catego- rized transfer learning scenarios into four groups based on the disparity between the original and downstream tasks, based on task objectives and data distributions (Figure T). We found that VPT is advantageous when (Case /) substantial gap exists between the original and downstream tasks (e.g., transitioning from classification to counting) or (Case 2) the data distributions are notably sim- ilar between the tasks (e.g., both deal with natural images). The size of the downstream task dataset also influences this preference, with FT becoming increasingly favorable as the data size grows.
We further investigate why VPT excels. While one plausible hypothesis suggests that FT is more prone to overfitting due to its numerous tunable parameters, our experiments reveal that this is only part of the story. Overfitting is primarily observed in Case (high task disparity), while in Case 2, both methods show trend for overfitting. A possible explanation is that the ad- ditional parameters introduced by VPT offer additional dimensions to escape the local minima of the pretrained model. However, empirical results do not support this assumption when we compare methods with additional dimensions. Our exploration of various tuning method variations underscores the importance of preserving the original feature for achieving superior performance, al- beit through a non-obvious pathway. Our results sug- gest that VPT preserves features and add parameters in unique manner that is pivotal in this process.
Data
Similar
Dissimilar
Count, Lecation, etc
Similar
Task
Dissamalar
Full fine tuning
Visual prompt tuning
Figure 1: VPT is identified to be prefer- able in 3 of transfer learning scenar- ios when downstream data is limited.
Overall, this study aims to provide a thorough reassessment of the effectiveness of VPT compared to FT. The primary contributions of this paper can be summarized as follows:
We identify the transfer learning scenarios where VPT proves advantageous by considering two critical dimensions: data distributions and task objectives. Notably, VPT is preferable in 3 out of quadrants, with FT gradually closing the performance gap as the downstream data size increases.
We uncover that overfitting alone does not account for all of VPT's success. Moreover, the ad- vantage of VPT cannot be attributed solely to additional parameters aiding the model in escaping local minima. The unique introduction of extra parameters plays key role VPT's performance.
To showcase the efficacy of prompt tuning over full finetuning, provide attention map visual- izations for both methods and demonstrate that visual prompts could enhance feature learning.
2 RELATED WORK
Full finetuning. The emergence of convolutional neural networks (CNNs) (e.g., VGG 166, ResNet [40], MobileNet [45) 90 44) makes a significant contribution to the rapid advancements in computer vision. The trained CNN backbones (knowledge) large-scale datasets can be easily transferred to adapt downstream vision tasks through finetuning (normally full finetuning), which termed as "pretrain-then- thent-innntnee"" This approach allows for a flexible adaptation, standing in
contrast to earlier methods that relied on hard-designed task-specific models 19 5 20). Instead of training each task from scratch, initializes the model with pre-trained weights, updating all the parameters from the original backbone with separate instances for different tasks. Full fine- tuning is a common practice under this paradigm, which keeps being powerful yet affordable approach [47] [34] 105) [3 with broad applicability. The subsequent prominent transformer based architectures in vision (e.g., VIT 24], Swin (66) inherited the same pipeline for adaptation as train- ing these backbones from scratch can hardly learn global features 24 12] on small datasets (i.e., Transformers extract features with less inductive bias, which contributes to continuously improve accuracy by increasing training data [26] 24). However, the limited training data can not fully ex- ploit the capacity). Recent breakthroughs
language for scaling up models size led to stronger generality [4] 136 83 6). Following this tendency, upcoming vision models (e.g., Florence [124], CoCa [23)) are heavily parameterized, when following the common approach under "pretrain-thes -thnn- finetune" paradigm, becomes impractical to fully finetune these models. This is primarily due the
2

---

Published as a conference paper at ICLR 2024
inherent inefffccetr-eeffcientt nature and the storage-wing expensive requirements of full finetuning. Therefore, parameter-efficee efficieet finetuning methods become a critical research area.
Visual Prompt Tuning. With the significant growth in the scale of current vision models, the devel- opment of parameter-effiient finetuning methods under preerriinthhen-inetmm paradigm becomes increasingly critical. Current methods be generally categorized into partial tuning [T6] 48 (9), extra module 88 129: 8 and prompt tuning [49] 50 23 118; 125], while partial tuning and ex- tra module face several limitations that hinder their application: First, they generally cannot reach 48
69:; Second, some require to insert specific ar- g during tuning, rendering them non-universal solutions when considering various backbones. Prompt tuning (59], on the other hand, provides a straightforward solution with strong performance gains during finetuning, which is originally proposed for language-
competitive results to full finetuning 49 16
chitecture/block design [29 85
domain (68) 42 63 [8]. The emergence of visual-related prompt tuning [66 30 [15] has only
recently come to the fore, signaling a new paradigm in parameter-efficient finetuning techniques in the field of computer vision. Generally, prompt tuning prepends extra sets of learnable parameters to the input sequence of backbones, and only updates these parameters during finetuning. While appearing to be simplistic, the paradigm of visual prompt tuning has exhibited satisfactory per- formance enhancements, as opposed to other parameter-efffcient finetuning methods which require specific module designs (i.e., extra module) [85
coercive freezing of certain portions of 69 Moreover, these methods show a substan- tial performance gap to full finetuning, thereby rendering their widespread implementation under "pretrain- fieen-fintuue" paradigm. Although current visual prompt tuning presents promising re- sults, it remains unexplored the comprehension of the underlying mechanisms that are responsible for their resounding success. In contrast, in the field of NLP, there are several works [13] 107 that investigate various aspects of prompt tuning (e.g., data scale, evaluation, stability). In light this view, this paper endeavors to explore and elucidate diverse facets pertaining to visual prompt tuning.
129 8
the backbone network (i.e., partial tuning) [16]
48
METHODOLOGY
Full finetuning is widely used finetuning method under "pretrain-thhe--fnetue paradigm. In particular, given a pre-trained model with parameter denoted by and downstream dataset D, full finetuning aims to learn a new set of parameters under the same model architecture as in pretraining. The objective of full finetuning is to minimize the corresponding loss function L(O') between the predictions [(x; 0') and the ground truth y by optimizing the parameters 0' over the entire network. While full finetuning obtains good performance with sufficient data, it is computationally expensive and requires the storage of all parameters.
Visual Prompt Tuning, in contrast, keeps the pretrained backbone model frozen while finetuning only a small set
* Frozen Tuned
of task-specific visual prompts 44 65
34 59 119 [08).
Ln
Ln
defined as P= {Po, Pi...., ......,,P PN-1) where P:-1 1 is learn- able vectors which are prepended to the input sequence of the i-th layer (as shown in Figure 2)b)). The optimiza- tion process of prompt tuning involves updating only the newly added prompts while keeping the pretrained model 0 fixed. As a consequence, prompt tuning is particularly beneficial in terms of limiting the overall parameter stor- age and updating, and preserving the learned knowledge without forgetting during the finetuning phase. In gen- eral, for a Vision Transformer (24) (ViT) with N layers L, prompt tuning can be formulated as:
L
L
L
L
(a) Full fine tuning
(b) Visual prompt tuning
Figure 2: Full finetuning us. visual prompt tuning. Visual prompt tuning only learns a small set of prompts.
Z = Li (Po, E).
Z, ( 7--1) 2 N, (1)
y = Head(Zn)
Z: is the contextual embeddings computed by the in encoder layer. E is the image patch embed- dings initialized with frozen embedding projection from the backbone. classification head is then used map the final layer's output embedding the prediction Trainable and frozen parameters are represented in different colors, respectively.
'The mapping of [CLSS token is general procedure in VIT. However, there are some vision transformer architectures (e.g., Swin (66), CaiT [TO1) do use (CT.S) and utilize glohal pooled Z input for Head.
3

---

Published as a conference paper at ICLR 2024
4
WHEN SHOULD WE CHOOSE VPT?
Our investigation starts with experimental analysis to identify the conditions that VPT are pre- ferrable. We first introduce our experiment settings and then present our key findings.
4.1 EXPERIMENT SETUP
Dataset Our experiments are conducted VTAB-IK [17 image classification benchmark, which encompasses 19 visual tasks categorized into three groups: Natural, Specialized, and Structured (i.e., for image examples, refer to $D). The Natural group consists of seven datasets comprising nat- ural images captured by standard cameras. The Specialized group includes four datasets that cover images taken by specialized equipment. The Structured group contains eight tasks that require ge- ometric comprehensions, such as counting and distance measurement. Unless stated otherwise, we follow the standard experimental setup of [49] 127 37 and use an 800-200 split for hyperparam- eter tuning, where 800 samples are used for training and 200 for validation. We evaluate the final performance on the full testing set. For the one-shot experiments, please refer to Appendix $ $ Baselines To ensure fair comparisons, we adopt the approach in VPT [49] and focus on the widely- used Vision Transformer (ViT) [24) for image classification. For the pre-training stage, we follow the default procedure outlined in 24 [6) and pre-train the vision Transformer
the ImageNet- 21 dataset [89]. Our implementation and experiments are conducted using the publicly available mode + It should be noted that there other backbone models (e.g., Swin (66)) and training ob- jectives (e.g., MAE (4I)) that can be used for this study, We have observed similar patterns.
Implementation Details Following 44 37 69 [5], conduct grid search match the best tun- ing hyperparameters, learning rate (i.e., [50, 25, 10, 5, 2.5, 1, 0.5, 0.25, 1, 0.05J). and weight decay (i.e.. [0.01, 0.001, 0.0001, 0.0]) on val set for each task. Notably, we follow (37 and do not cover specific-designed large learning rate in 40 for general propose. In the training of all models, a cosine decay policy is employed to schedule the learning rate. The total number training epochs is set to be 100 (including 10 warm-up epochs). We follow the same batch size setting: 64/128 for ViT-B/16. Our method is implemented in Pytorch [80]. Experiments are conducted on NVIDIA A100-80GB GPUs. For reproducibility, our full implementation will be publicly released.
4.2 INITIAL EXPERIMENTAL RESULTS
The initial results of FT and VPT are presented in the first two rows of Table I and Table 2 (i.e., the last two rows: Mixed and PT-then-PT, will introduced and compared FT and VPT $5.2). Out of all 19 datasets/tasks, VPT performs better on 16 instances, including 6/7 Natural datasets, 2/4 Specialized datasets, and 8/8 Structured datasets.
Table Image classification accuracy on various training strategies on VTAB-1k [12]] Natural
and Specialized for ViT-B/16 [24] pretrained supervised ImageNet-21k 89] Underlines indicate the better results between FT and VPT, and bolds indicates the best results among all variants. We
report five runs test average accuracy instead of three 49
to consider more randomness, and
"Number of Wins" in (-) compared to full finetuning (Full)
47
VIT-B/16 24
VIAB-IK 122
Nateral [?]
VIAB-IK (12)
Specialized (4)
(85.8M)
CIFAR- 100 Caltech101 DTD Flowers 102 Pets SVHN Sun399
Mean
Putch Camelyon Eu Resise45 Retinopathy
Mcan
27
88.1
65.0
84.6
87.3
39.5
75.19
81.2
95.7
83.2
83.48
VPT 49
77.7
90.2
68.8
98.1
88.4
82.5
51.0
79.53(6)
77.9
96.2/9)
83.4
73.1
82,6(()
Mixed
61.5
86.2
60.2
95.6
85.0
85.7
35.4
72.80(1)
77.8
95.2(7)
74.0
71.0
79.78(1)
PT-then-PT
85.9
89.0
66.3
86.3
87.1
36.9
(8.4))4)
79.4
95.6
81.8
74.0
82,70(1)
Table Image classification accuracy on various training strategies on VTAB-1k 127 Struc-
tured for ViT-B/16 24] pretrained on supervised ImageNet-21k (8).
VIT-B/16 [24
VTAB-Ik [127] Structured [8]
Mean
(85.8M)
Clevrfent. Clevrídist. DMLab KiTTIkist. dSprites/loc, dSprites/ori. SmalINORB/az. SmallNORB/elev,
FT 47
55.4
58.2(2)
40.4
74.7
54.0
47.0
26.2
28.6
48.07
VPT 49
68.2
58,2(3)
45.3
77.7
78.4
48.4
31.2
41.3
56.09(8)
Mixed
54.6
20.9
45.2
28.5
42.94(3
FT-then-PT
63.6
61.3
39.1
73.0
55.1
51.6
27.6
31.7
50.38(6)
hhtg:://gghhhh.cco/ooooo/oeeeeee transformer
4

---

Published as a conference paper at ICLR 2024
4.3 CHOOSE VPT FOR THREE QUADRANTS OF TRANSFER LEARNING
Upon initial examination, these results do not re- veal distinct pattern indicating when VPT out- performs FT. However, upon deeper reflection on the nature of transfer learning, the effectiveness of finetuning methods under the "pretrain-then- finetune" paradigm can be significantly impacted by the disparities between pretraining and fine- tuning. These disparities can be attributed to two key aspects: 1) the data distributions in the pretraining and finetuning datasets are dissimilar, and 2) the nature of the downstream task objec- tive is different from the pretraining one.
Full fine tuning
Low task disparity
Prompt tuning
High task disparity
300
275
250
FID Score
225
200
175
150
125
Hence we analyze the datasets by categorizing them into four quadrants along these two dimen- sions. To measure the discrepancy in image dis- tribution, we adopt the Fréchet Inception Dis- tance (FID) [17 (66), which is known to corre- spond well with human judgments of image qual- ity and diversity [I66 [7]. In terms of task dis- parity, tasks in the Natural and Specialized sub- sets are closely related to image classification and thus have low disparities, while tasks in Struc- tured, such as counting and distance, are regarded as distinct from image classification.
100
CIFA100
Catech101
DTD
Flowers100 200
Rets
SVHN
Sun397
Patch Camelyon
EUrOSAT
Restsc45
netinopathy
Ctevneett
DMIeb
KITTiiott.
Spritesfoo.
SmallNORB
SmallNORBIOCK
VTAB-IK Beachmark
Figure 3: Overall FID score with respect to win/loss of visual prompt tuning on VTAB-1k benchmark categorized into Natural, Special- ized and Structured, respectively. represent the method with higher accuracy. Un- der the same train-val split and low task dispar- ity, higher FID score might potentially lead to relatively higher accuracy on full finctuning, and vice versa. Solid filled represents that the disparity between the target task and the pre- trained task is small while slash filled means large disparity (e.g.. distance, azimuth, count- ing). The FID scores show significant robust- ness in repeat runs (i.e., std < 0.5%), we there- fore do not present error bars here.
colors
The FID scores with respect to the better fine- tuning method of all 19 tasks under the default train-val split are shown in Figure 3] The fig- ure shows that VPT reaches superior performance than FT in two different scenarios: (1) when the disparity between the task objectives of the origi- nal and downstream tasks is high, or (2) when the data distributions are similar. Only for those tasks with low disparity (the left 11), a relatively larger FID score (dissimilar datasets) typically leads to higher performance of full finetuning compared to prompt tuning
4.4 GAP NARROWS AS DOWNSTREAM DATASETS EXPAND
The above observation holds for scenarios when we have a limited amount of data for the down- stream tasks. However, the effectiveness of different finetuning methods may vary high-resource scenarios. To investigate this, we conducted comprehensive evaluation of FT and VPT across different tuning set sizes. Specifically, we re-create the training and validation sets that keep a con- sistent 4:1 ratio, and vary the number of training samples from 400 to 20,000 to explore the behavior of the FT and VPT when the tuning data size grows. is important to note that not all datasets support all combinations of splits due to insufficient data samples (e.g.. VTAB- Ik Natural Oxford Flowers 102 73 has only 1020 images for both ra in and va
The performance of full finetuning and prompt tuning across different training set sizes and datasets is presented in FFigure 4 The figure shows that the performance gap between full finetuning and prompt tuning decreases the amount of training data increases, with full finetuning even outper- forming prompt tuning under high-resource settings. These results demonstrate the robustness and generality of prompt tuning in the face of limited finetuning examples. Although full finetuning generally achieves higher accuracy when rich data examples are available, the prompt tuning still
'ere the range of FID scores reported appears to be larger than existing approaches 999 17 79 128
This is because current methods typically report the distance between real and generated images, whereas our analysis involves the computation of distances hetween real datasets with significantly different distributions.
5

---

Published as a conference paper at ICLR 2024
100
100
100
90
80
90
Accureyy
60
80
50
50
40
Promat
C19A0200
70-
Prompt
Natch Carreyyon
30
Promer
Pets
EurOeAT
Saik
20
Sundg)
Aptroopattyy
Sma INORSOO
20
60
Log-scale number of training set sumples
Figure 4: Analysis of dataset capacity on VTAB- Ik Natural (left), Specialized (middle) and Struc- tured (right), respectively. For each group, we select four datasets and plot accuracy plots on FT (i.e., solid lines) and VPT (i.e., dotted lines). We take the log scale of training data samples for better separation and each color stands for an individual classification task. Each point is given by average over five runs. In general, with the dataset increasing in size, the performance gap between FT and VPT becomes narrow. FT even surpasses VPT in 9 data samples (the same tendency takes place in other datasets). For per-task accuracy tables among different dataset scales and detailed FT and VPT accuracy plots, see the supplementary material $A
cases in this plot with the increasing of
Table 3: One-shot image classification accuracy on VTAB-Ik [127] Natural and Specialized for VIT-B/16 (24] pretrained supervised ImageNet-2Ik 89 I. Experiments show that VPT outper- forms the FT in 13 of 19 instances under the one-shot learning setting, which further suggests the advantage of VPT in limited data cases. "Tuned/Total" is the average percentage of tuned parameters
required by 19 tasks.
VIT-B/16 24
VTAB-IK [12] Natural [7]
VTAB-IK 127 Specialized [4]
(85.8M)
CIFAR-100 100 Caltech101 DTD Flowers102
Pets
SVHN Sun397
Patch Camelyon EuroSAT Ressse15 Retinopathy
Mean
FT 47
37.4
67.9
25.3
80.2
37.8
12.7
30.3
41.66
SM.2
44.1
42.3
32.6
44.30
37.4
70.5
11.6
47,45(3)
Tused/ / Total
0.20%
0.20%
0.15%
0.10%
0.04%
0.54%
0.41%
0.23%
1.06%
1.0%%
0.15%
0.02%
0.57%
reaches a competitive performance with much fewer parameters. This observation aligns with cur- rent research in NI.P prompt tuning [33 22 and suggests that prompt tuning techniques have great potential in both low- and high-resource scenarios.
To further test the effectiveness of the two methods in low-resource settings, conducted one-shot learning experiments where each class is learned from single labeled example [104, [8] These experiments represent an extreme case in which novel tasks after having been exposed to only a small amount of training data. Table 3 and Table 4 show the experimental results, with prompt tuning outperforming full finetuning in 13 out of 19 tasks and achieving substantially higher accuracy in some cases (i.e., 41 .66% us 56.94% % in VTAB-1k Natural). These results favor our assumption that prompt tuning more effective than full finetuning for the aforementioned three quadrants when only limited finetuning examples are available. We observe that only half of the Structured cases, VPT performs better. Overall, the accuracies for one-shot experiments are comparatively low. Hence it should be noted that randomness could play a role in shot experiments, and the results might not reliably conclusive.
seek to evaluate the methods' ability to adapt to
5
WHY DOES VPT OUTPERFORM FT?
5.1
OVERFITTING IS PART OF THE REASON
After identifying the conditions that VPT would be favorable over FT, we dive deeper to explore the underlying reasons. One hypothesis is that FT optimizes all the parameters in the backbone, which may easily overfit to the downstream task, especially when limited data is available [22 21 39].
Figure 5 presents training and testing loss curves, up 100 epochs, for six representative tasks in
VTAB-Ik [17). comprising of 2 tasks for from each kind (Natural, Specialized, and Structured). Comprehensive results covering all 19 tasks available in the supplementary material SA
We observe that for 8 out of Structured tasks, both FT and VPT overfit (i.e., testing error increases training progresses). These tasks, such counting and angular measurement, are very different

---

Published as a conference paper at ICLR 2024
Table One-shot image classification accuracy on VTAB-Ik [127]
Structured for ViT-B/16
pretrained on supervised ImageNet-21K [89].
VIT-B/16 24
VIAB-II
Structured [8]
(85.8M)
Clevrlent. Clevs/dit.. DMLab KITTI/dist. dSprites/loc dSprites/or. SmalNNORB/az SmallNORB/elex
Mean
FT
17.1
40.5
7.4
17.11
VPT
21.5
22.1
18.7
37.0
6.2
14.9
16.834)
Tuned/ / Total
0.54%
2.11%
1.7%
0.54%
0.12%
0.55%
2.12%
2.11%
1.14%
Full fine-tuning
Prompt tuning
Mixed
FT-then-PP PT
Coltech101
Pets
Training Loss
Testing Loss
Training Loss
Testing Loss
100
20
102
Epochs
Epochs
Epochs
Epochs
EuroSAT
Resisc45
Training Loss
Testing Loss
Training Loss
Testing Loss
100
Epoclis
Epochs
Epochs
Epochs
Clevr/count
SmallNNORB/azimuch
Training Loss
Tosting Loss
Training Loss
Testing Loss
2-
Epochs
Epochs
Epochs
Epochs
colors represent four training strategies: full finetuning, prompt tuning, mixed, and FT-then-PT, respectively, We show two representative tasks from VTAB-IK 1k Natural, Specialized, and Structured, respectively. Full results and log scale results are presented the supplementary material A
Figure 5: Training/testing loss curves of six datasets from VTAB-IK.
from image classification, and may require more training data for model adaptation. We conducted additional experiments by enlarging the training set (to 5000 and 10000) of these tasks in Fiiure 6 can be observed that with the increase of training samples, the phenomenon of overfitting is mitigated in both methods.
However, among the remaining 10 tasks within the categories of Natural and Specialized, over- fitting behavior is observed in only task. In other words, training and testing losses consistently decrease for both methods in scenarios where the task objectives are similar between the original and downstream tasks. In such cases, overfitting is the cause of performance degradation of FT.
5.2
OPTIMIZATION DOES NOT PARTICULARLY FAVOR VPT
Another possible explanation for why prompt tuning achieve superior performance is that it is the additional dimensions introduced by the prompts that help the loss to be further optimized during the tuning process. This hypothesis is illustrated by Figure
To test our hypothesis in depth, we perform experiments on two additional variants of tuning strate- gies besides full finetuning and prompt tuning. The first method is noted Mixed, where we add visual prompts and perform full finetuning on the entire network instead of focusing only on net- work parameters prompt vectors in isolation. The second method is called FT-then-PT, where

---

Published as a conference paper at ICLR 2024
first perform full finetuming the network for the downstream task and then prepend additional prompts and perform prompt tuning for the same task. This approach applies 200 epochs in total.
We compare the above methods with full fine-
Full fine-tuning
Training loss
tuning and visual prompt tuning in Figure 5 Per-task performance is also detailed in Table and Table 2] We observe that in general, VPT
Prompt tuning
Testing loss
Patch Camelyon (5000)
Patch Camelyon (10900)
Lovs
Loss
demonstrates the highest performance. The FT
approach and the FT-then-P7 approach yield similar results, albeit with a noticeable gap compared to VPT. In contrast, the Mixed ap- proach exhibits the lowest performance among the evaluated methods.
Cleveient, (5000)
Clevrient, (10000)
Loss
These results indicate several key findings. (1) The presence of additional dimensions does not significantly aid the optimization process in es- caping local minima. If this were the case, we would expect either the Mixed or the FT-then- PT method to outperform FT, but this is not observed. (2) Preserving the original feature down-
SmalINORBlelev. (5000)
SmalINORB/elev, (10000)
Eepochs
Epachs
Training-hesting loss curve for VTAB-Ik overfitting datasets
proves be crucial for transferring stream tasks, as evidenced by the superior per- formance of VPT compared to FT-then-PT. (3) Maintaining a fixed feature space in VPT may compel it learn more valuable embeddings with the trainable prompts, as it significantly outperforms the Mixed method. These obser- vations lead to an important insight: pretrained parameters play a pivotal role in capturing gen- eral features, while the added parameters are important for potentially encoding task infor- mation in the context of transfer learning.
Figure 6: Training/testing loss curves for VTAB-1k Structured datasets with larger train- ing sets. = colors represent full finetuning and prompt tuning, respectively. Solid lines are the training loss curves and dashed lines show the cor- responding testing curves. We select three tasks that suffer from overfitting and enlarge the dataset train-val split 5000- (left) and 10000-2500 (right), respectively. Full results are shown in the supplementary material A
5.3 FURTHER OBSERVATIONS
Overall, our observations indicate that the preservation of initial features is important for VPT's success, but in a very sophisticated manner. Notably, VPT surpasses more straightforward methods of feature preservation, such as retraining the final linear probe layer or incorporating multi-layer probe (37), even though all these meth- ods maintain the originally learned features.
This potentially explains cases in which data distributions exhibit similarity, but not scenarios characterized by distinct data distri- butions and high task disparity. In these instances, we hypothesize that the substantial task divergence requires a stronger need for ad- ditional finetuning examples achieve comprehensive adaptation, whereas prompt tuning demonstrates better generalization. The in- troduced prompts serve as guiding mechanisms for the model enhance task-specific feature learning, pivotal aspect of task en- coding, particularly in situations with limited finetuning data.
Figure 7: Hypothesis of extra dimensions helping the pre- trained model to be further optimized, e.g., by escaping local minima. In this illus- tration, pre- training the orig- inal ID function (the red line) might get stuck in the middle (the red star). When a second dimension is introduced dur- ing finetuning, is casier to be better optimized (from the yellow line to the yellow star). However, found that this hypothesis does not hold.
It's also worth discussing scenarios in which FT outperforms VPT, specifically in situations involving similar tasks with varying data distributions. Our hypothesis is that the feature representation cap- tured by the pretrained model may not be well-suited for the down- stream task due to significant differences in data distribution. Con- sequently, updating all model parameters through full finetuning be- comes more effective for learning the feature representation within
8

---

Published as a conference paper at ICLR 2024
the new distribution. Conversely, when finetuning data is similar, there may be no need to modify the backbone model extensively; instead, a minimal set of prompts can be inserted for adaptation.
5.4 VISUALIZING THE EFFECT OF VPT ON FEATURE LEARNING
To the best of our knowledge, the visual explanations of visual prompt tuning are found to be rare [60| 75 71. In light of this view, we seek to investigate whether prompt tuning can offer actual visualization meanings and support stronger visualization explanations, particularly in cases apply
where it outperforms full finetuning. In this experiment,
class activation mapping (GradCAM) ) [92], which is a popular technique for producing visual explanations for deci- sions 10 2 35 29 II In general, it utilizes the gradients of target concept and channels them through the final layer of the network to generate a coarse localization map that accentuates the salient regions in the image which contribute significantly towards predicting the concept of interest.
Figure 8 presents examples that full finetuning fails to recognize while prompt tuning makes correct classification during testing. can be seen from the results that a clear concept of interest can be observed from both methods through heat map. In these prompt tuning success cases, we can see straightforward vi- sual explanation differences (e.g., take the bi- cycle image Figure upper right as an ex- ample, when full finetuning fails to make cor- rect decision, prompt tuning instead recognize the bicycle with its structural features success- fully), which suggests that prompt tuning ef- fectively guides the model to pay more atten- tion to the important regions of the images, and thus is capable of enhancing the learning of stronger features by leveraging domain-spectie information and patterns that might not be well captured in full finetuning. An additional vi- sual explanation technique (i.e., Integrated Gra- dients (100) and more visual inspection examples GradCAM are also included in the supplemen-
Ingut
Prompe
Ingut
Prompe
Figure 8: Visual inspection of full finetuning and prompt tuning using GradCAM [92]. Note that red regions correspond high score for the class. From left to right are input image after stan- dard data augmentation, GradCAM results for full finetuning and GradCAM results for prompt tun- ing. Figure best viewed in color.
tary material B
6 CONCLUSION AND DISCUSSION
When models scale up, how shall we face the elephant in the room? Driven by this question, we aim to provide a justifiable recommendation regarding the choice between full finetuning or prompt tun- ing during training. In this pursuit, we conduct an extensive study across 19 datasets, comparing full finetuning with visual prompt tuning, to examine various hypotheses regarding the effectiveness of visual prompt tuning. Our experimental results show that overfitting is not the root cause of full fine- tuning's 's inferior performance to prompt tuning. We further notice that dataset disparities might be a potential factor in the behavior of full finetuning and prompt tuning. Also, prompt tuning shows its strong generality with limited data, while full finetuning catches up or surpasses prompt tuning with more data presented. Attention map visualization suggests that the visual prompts are capable of enhancing the learning of features that might not be well captured in full finetuning. We suggest that prompt tuning should be applied consciously under "pretrain- www--tthenn--teeeeeeee paradigm. Task dispar- ities and dataset scale are two main factors that determine the most suitable way for downstream finetuning. In spite of our comprehensive coverage of various datasets in image recognition, is noteworthy that other visual-related tasks for finetuning (e.g.. semantic segmentation [62] (8) have not been fully explored. This observation motivates us for future investigation of such scenarios.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
To help readers reproduce our results, we have described the implementation details in $4.1 and $ We will release our source code after acceptance. All the datasets use are publicly available.
ACKNOWLEDGMENT
This research was supported by the National Science Foundation under Grant No. 2242243.
REFERENCES
[1] Sercan Ô Arik and Tomas Pfister. Tabnet: Attentive interpretable tabular learning. In AAAI, 2021.
[2] Alejandro Barredo Arrieta, Natalia Rodrigudrr Javier Del Ser, Adrien Bennetot, Siham Tabik, Alberto Barbado, Salvador García, Sergio Gil-López. Daniel Molina, Richard Ben- jamins, et al. Explainable artificial intelligence (xai): Concepts, taxonomies, opportunities and challenges toward responsible ai. Information Fusion, 58, 2020.
[3] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Visual prompting: Modifying pixel space to adapt pre-trained models. arXiv preprint arXiv:2203. 17274, 2022.
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bobg. Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021.
[5] Leo Breiman. Random forests. Machine learning, 45, 2001.
[6] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In NeurIPS, 2020.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In CVPR, 2018.
[8] Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory, not parameters for efficient on-device learning. In NeurIPS, 2020.
[9] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, 2021.
[10] Diogo V Carvalho, Eduardo M Pereira, and Jaime Cardoso. Machine learning interpretabil- ity: A survey on methods and metrics. Electronics, 8(8):832, 2019.
[11] Hila Chefer, Shir Gur, and Lior Wolf. Transformer interpretability beyond attention visual- ization. In CVPR, 2021.
[12] Bin Chen, Ran Wang. Di Ming, and Xin Feng. Vit-p: Rethinking data-efficient vision trans- formers from locality. arXiv preprint arXiv:2003.02358, 2022.
[13] Guanzheng Chen, Fangyu Liu, Zaiqiao Meng, and Shangsong Liang. Revisiting parameter- efficient tuning: Are we really there yet? arXiv preprint urXiv:2202.07966 2022.
[14] Shoufa Chen, Chongjian Ge, Zhan Tong, Jiangliu Wang. Yibing Song, Jue Wang, and Ping Luo. Adaptformer: Adapting vision transformers for scalable visual recognition. arXiv preprint arXiv:2205.13533 2022.
[15] Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In ACM Web Conference, 2022.
10

---

Published as a conference paper at ICLR 2024
[16] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self self-supervised vision transformers. In ICCV, 2021.
[17] Min Jin Chong and David Forsyth. Effectively unbiased fid and inception score and where to find them. In CVPR, 2020.
[18] Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. What does bert look at? an analysis of bert's attention. arXiv preprint arXiv:1906.04344 2019.
[19] Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20, 1995.
[20] Thomas Cover and Peter Hart. Nearest neighbor pattern classification. IEEE Transactions on Information Theory, 13(1), 1967.
[21] Tom Dietterich. Overfitting and undercomputing in machine learning. ACM Computing Surveys, 27(3), 1995.
[22] Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 2023.
[23] Shaohua Dong. Yunhe Feng, Qing Yang. Yan Huang, Dongfang Liu, and Heng Fan. Efficient multimodal semantic segmentation via dual-prompt learning. arXiv preprint arXiv:2312.03360, 2023.
[24] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
[25] Frédéric Dufaux. Grand challenges in image processing, 2021.
[26] Stéphane d' Asool,,, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Lev- ent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In ICML, 2021.
[27] Dumitru Erhan, Yoshua Bengio, Aaron Courville, and Pascal Vincent. Visualizing higher- layer features of deep network. University of Montreal, 2009.
[28] Marc Fischer, Alexander Bartler, and Bin Yang. Prompt tuning for parameter-efficient medi- cal image segmentation. arXiv preprint arXiv22211.09233, 2022.
[29] Shang-Hua Gao, Ming-Ming Cheng, Kai Zhao, Xin-Yt Zhang, Ming- Yang, and Philip Torr. Res2net: A new multi-scale backbone architecture. IEEE TPAMI, 2019.
[30] Yunhe Gao, Xingjian Shi, Yi Zhu, Hao Wang, Zhiqiang Tang, Xiong Zhou, Mu Li, and Dimitris N Metaxas. Visual prompt tuning for test-time domain adaptation. arXiv preprint arXiv:2210.04833 2022.
[31] Timnit Gebru, Jonathan Krause, Yilun Wang. Duyun Chen, Jia Deng, and Li Fei-Fei. Fine- grained car detection for visual census estimation. In AAAI, 2017.
[32] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, 2012.
[33] Yolanda Gil, Daniel Garijo, Deborah Khider, Craig A Knoblock, Varun Ratnakar, Maximil- iano Osorio, Hernán Vargas, Minh Pham, Jay Pujara, Basel Shbita, et al. Artificial intelligence for modeling complex systems: taming the complexity of expert models to improve decision making, ACM Transactions on Interactive Intelligent Systems, 2021.
[34] Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. Ppt: Pre-trained prompt tuning for few-shot learning. arXiv preprint 1rXiv:2109.0445 2021.

---

Published as a conference paper at ICLR 2024
[35] Riccardo Guidotti, Anna Monreale, Salvatore Ruggieri. Franco Turini, Fosca Giannotti, and Dino Pedreschi. A survey of methods for explaining black box models. ACM Computing Surveys, 51(5), 2018.
[36] Meng-Hau Mang--Hao Guo, Cheng Ze Lu, Zheng-Ning Liu, Ming-Ming Cheng, and Shi-Min Hu. Visual attention network. arXiv preprint arXiv:2202.09741, 2022.
[37] Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang, Siyuan Qi, and Dong- fang Liu. E' 2vpt: An effective and efficient approach for visual prompt tuning. In ICCV, 2023.
[38] Antoine I. Harfouche, Daniel A Jacobson, David Kainer, Jonathon C Romero, Antoine H Har- fouche, Giuseppe Scarascia Mugnozza, Menachem Moshelion, Gerald A Tuskan, Joost JB Keurentjes, and Arie Altman. Accelcrating climate resilient plant breeding by applying next- generation artificial intelligence. Trends in Biotechnology. 37(11), 1), 2019.
[39] Douglas M Hawkins. The problem of overfitting. Journal of Chemical Information and Computer Sciences, 44(1), 2004.
[40] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
[41] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In CVPR, 2022.
[42] Yun He, Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, YaGuang Li, Zhao Chen, Donald Metzler, et al. Hyperprompt: Prompt-based task-conditioning of transformers. In ICML, 2022.
[43] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochre- iter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurlPS, 2017.
[44] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chied Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, al. Searching for mobilenetv3. In ICCV, 2019.
[45] Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andrectto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: 1704.0466,. 2017.
[46] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint (at::2206.99000 2021.
[47] Eugenia lofinova, Alexandra Peste, Mark Kurtz, and Dan Alistarh. How well sparse imagenet models transfer? In CVPR, 2022.
[48] Menglin Jia, Zuxuan Wu, Austin Reiter, Claire Cardie, Serge Belongie, and Ser-Nam Lim. Exploring visual engagement signals for representation learning. In ICCV, 2021.
[49] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariha- ran, and Ser-Nam Lim. Visual prompt tuning, In ECCV, 2022.
[50] Chen Ju, Tengda Han, Kunhao Zheng, Ya Zhang, and Weidi Xie. Prompting visual-language models for efficient video understanding. In ECCV, 2022.
[51] Aditya Khosla, Nityananda Jayadevaprakash, Bangpeng Yao, and Fei-Fei Li. Novel dataset for fine-grained image categorization: Stanford dogs. In CVPR Workshop, 2011.
[52] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In ICMI., 2017.
12

---

Published as a conference paper at ICLR 2024
[53] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In ECCV, 2020.
[54] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, 2009.
[55] Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJCV, 2020.
[56] Tuomas Kynkäänniemi, Tero Karras, Mika Aittala, Timo Aila, and Jaakko Lehtinen. The role of imagenet classes in fr', 'echet inception distance. In ICLR, 2023.
[57] Thibault Laugel, Marie-Jeanne Lesot. Christophe Marsala, Xavier Renard, and Marcin De- tyniecki. The dangers of post-hoe interpretability: Unjustified counterfactual explanations. In IJCAI, 2019.
[58] Franck Le, Mudhakar Srivatsa, Raghu Ganti, and Vyas Sekar. Rethinking data-driven net- working with foundation models: challenges and opportunities. In ACM Workshop, 2022.
[59] Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP, 2021.
[60] Aodi Li, Liansheng Zhuang, Shuo Fan, and Shafei Wang. Learning common and specific visual prompts for domain generalization. In ACCV, 2022.
[61] Pantelis Linardatos, Vasilis Papastefanopoulos, and Sotiris Kotsiantis. Explainable ai: A review of machine learning interpretability methods. Entropy, 23(1), 2020.
[62] Lingbo Liu, Bruce XB Yu, Jianlong Chang, Qi Tian, and Chang- Wen Chen. Prompt-matched semantic segmentation. arXiv preprint arXiv:2208. 10/59, 2022.
[63] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ACM Computing Surveys, 2023.
[64] Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint 1rXiv:2110007602, 2021.
[65] Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable fine-tuning across scales and tasks. In ACL, 2022.
[66] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[67] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In NeurIPS, 2017.
[68] Fang Ma, Chen Zhang, Lei Ren, Jingang Wang, Qifan Wang, Wei Wu, Xiaojun Quan, and Dawei Song. Xprompt: Exploring the extreme of prompt tuning. In EMNLP, 2022.
[69] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yix- uan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018
[70] Aravindh Mahendran and Andrea Vedaldi. Understanding deep image representations by inverting them. In CVPR, 2015.
13

---

Published as a conference paper at ICLR 2024
[71] Chengzhi Mao, Revant Teotia, Amrutha Sundar. Sachit Menon, Junfeng Yang. Xin Wang, and Carl Vondrick. Doubly right object recognition: A why prompt for visual rationales. arXiv preprint arXlv:2212.06202, 2022.
[72] John Merrill, Geoff Ward, Sean Kamkar, Jay Budzik, and Douglas Merrill. Generalized integrated gradients: A practical method for explaining diverse ensembles. arXiv preprint vrXiv:1909.01888 2019.
[73] M-E. Nilsback and Zisserman. Automated flower classification over large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, 2008.
[74] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference Computer Vision, Graphics & Image Processing, 2008.
[75] Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTaek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, and Kyungwoo Song. Blackvip: Black-box visual prompting for robust transfer learning. arXiv preprint MxXix:3303.1477: 2023.
[76] Samet Oymak, Ankit Singh Rawat, Mahdi Soltanolkotabi, and Christos Thrampoulidis. On the role of attention in prompt-tuning. arXiv preprint arXiv:2306. 03435, 2023.
[77] Gerhard Paaß and Sven Giesselbach. Foundation models for natural language processing- pre-trained language models integrating media. arXiv preprint 27X 2023.
[78] Geondo Park, June Yong Yang, Sung Ju Hwang, and Eunho Yang. Attribution preservation in network compression for reliable network interpretation. In NeurlPS, 2020.
[79] Gaurav Parmar, Richard Zhang, and Jun- Yan Zhu. On aliased resizing and surprising sub- deties in gan evaluation. In CVPR, 2022.
[80] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-ppeformance deep learning library. In NeurIPS, 2019.
[81] Zheyun Qin, Cheng Han, Qifan Wang, Xiushan Nie, Yilong Yin, and Xiankai Lu. Unified 3d segmenter prototypical classifiers. In NeurlPS, 2023.
[82] Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. Pre- trained models for natural language processing: A survey. Science China Technological Sci- ences, 2020.
[83] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 2020.
[84] Sahana Ramnath, Preksha Nema, Deep Sahni, and Mitesh M Khapra. Towards interpreting bert for reading comprehension based qa. arXiv preprint arXiv:2010.08983, 2020.
[85] Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual do- mains with residual adapters. In NewrIPS, 2017.
[86] Tal Ridnik, Emanuel Ben-Baruch Asaf Noy, and Lihi Zelnik-Manor Imagenet-211 pretrain- ing for the masses. arXiv preprint arXiv:2104 / 10972 2021.
[87] Cynthia Rudin. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 1(5):206-215 2019.
[88] Cynthia Rudin, Chaofan Chen, Zhi Chen, Haiyang Huang, Lesia Semenova, and Chudi Zhong. Interpretable machine learning: Fundamental principles and 10 grand challenges. Statistic Surveys, 16:1-85, 2022.
14

---

Published as a conference paper at ICLR 2024
[89] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Fei- Fei Li. Imagenet large scale visual recognition challenge. IJCV, 2015.
[90] Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chieh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In CVPR, 2018.
[91] Maximilian Seitzer. pytorch-fid: FID Score for PyTorch. https://github: com/
mseitzer/pytorree fid, August 2020. Version 0.3.0.
[92] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient- based localization. In ICCV, 2017.
[93] Bhavin J Shastri, Alexander N Tait, Thomas Ferreira de Lima, Wolfram HP Pernice, Harish Bhaskaran, C David Wright, and Paul R Prucnal. Photonics for artificial intelligence and neuromorphic computing. Nature Photonics, 15(2), 2021.
[94] Baifeng Shi, Siyu Gai, Trevor Darrell, and Xin Wang. Toast: Transfer learning via attention steering. arXiv preprint arXiv:2305. 15542, 2023.
[95] Avanti Shrikumar, Peyton Greenside, and Anshul Kundaje. Learning important features through propagating activation differences. In ICML, 2017.
[96] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[97] Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman. Deep inside convolutional networks: Visualising image classification models and saliency maps. arXiv preprint arXiv:1312.6034, 2013.
[98] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. NeurlPS, 2017.
[99] Kihyuk Sohn, Yuan Hao, José Lezama, Luisa Polania, Huiwen Chang, Han Zhang, Irfan Essa, and Lu Jiang. Visual prompt tuning for generative transfer learning. arXiv preprint arXiv:2210.00900 2022.
[100] Mukund Sundararajan, Ankur Taly, and Qiqi Yan. Axiomatic attribution for deep networks. In ICML, 2017.
[101] Hugo Touvron, Matthieu Cord, Alexandre Sablayrolles, Gabriel Synnaeve, and Hervé Jégou. Going deeper with image transformers. In ICCV, 2021.
[102] Lifu Tu, Caiming Xiong, and Yingbo Zhou. Prompt-tuning can much better than fine- tuning cross-lingual understanding with multilingual language models. arXiv preprint arXiv:2210. 12:.6000.. 2022.
[103] Grant Van Horn, Steve Branson, Ryan Farrell, Scott Haber, Jessie Barry, Panos Ipeirotis, Pietro Perona, and Serge Belongie. Building bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grat fine-grained dataset collection. In CVPR, 2015.
[104] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. NeurlPS, 2016.
[105] David Vos, Till Döhmen, and Sebastian Schelter. Towards parameter-efficien automation of data wrangling tasks with prefix-tuning In NeurIPS Workshop, 2022.
[106] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-201 dataset. 2011.
[107] Chaozheng Wang, Yuanhang Yang, Cuiyun Gao, Yun Peng, Hongyu Zhang, and Michael R Lyu. No more fine-tum toning?nng? an experimental evaluation of prompt tuning in code intelligence. In ACM ESEC/FSE, 2022.
15

---

Published as a conference paper at ICLR 2024
[108] Qifan Wang, Yuning Mao, Jingang Wang, Hanchao Yu, Shaoliang Nie, Sinong Wang, Fuli Feng. Lifu Huang. Xiaojun Quan, Zenglin Xu, et al. Aprompt: Attention prompt tuning for efficient adaptation of pre-trainn pre-trained language models. In EMNLP, 2023.
[109] Wenguan Wang, Cheng Han, Tianfei Zhou, and Dongfang Liu. Visual recognition with deep nearest centroids. In ICLR, 2023.
[110] Wenxuan Wang, Jiachen Shen, Chen Chen, Jianbo Jiao, Yan Zhang, Shanshan Song, and Jiangyun Li. Med-tuning: Exploring parameter-effiient transfer learning for medical volu- metric segmentation. arXiv preprint arXiv:2334. 2023.
[111] Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. Multitask prompt tuning enables parameter-efficient transfer learning. arXiv preprint 2Xiv:2303302861,, 2023.
[112] Jingyuan Wen, Yutian Luo, Nanyi Fei, Guoxing Yang, Zhiwu Lu, Hao Jiang, Jie Jiang, and Zhao Cao. Visual prompt tuning for few-shot text classification. In ICCL. 2022.
[113] Walter F Wiggins and Ali S Tejani. On the opportumities and risks of foundation models for natural language processing in radiology. Radiology: Artificial Intelligence, 4(4), 2022.
[114] Cheng-En Wu, Yu Tian, Haichao Yu, Heng Wang, Pedro Morgado, Yu Hen Hu, and Linjie Yang. Why is prompt tuning for vision-language models robust to noisy labels? In ICCV, 2023.
[115] Yinghui Xing, Qirui Wu, De Cheng, Shizhou Zhang, Guoqiang Liang, and Yanning Zhang. Class- visual prompt tuning for vision-language pre-trained model. arXiv preprint arXiv:2208.0833 2022.
[116] Qiantong Xu, Gao Huang, Yang Yuan, Chuan Guo, Yu Sun, Felix Wu, and Kilian Weinberger. An empirical study on evaluation metries of generative adversarial networks. arXiv preprint arXiv::8066 00Xiv:/806.07755, 2018.
[117] Wanqi Xue and Wei Wang. One-shot image classification by learning to restore prototypes. In AAAI, 2020.
[118] Liqi Yan, Cheng Han, Zenglin Xu, Dongfang Liu, and Qifan Wang. Prompt learns prompt: exploring knowledge-awwe aware generative prompt collaboration for video captioning. In IJCAI, 2023.
[119] Li Yang, Qifan Wang, Jingang Wang, Xiaojun Quan, Fuli Feng, Yu Chen, Madian Khabsa, Sinong Wang, Zenglin Xu, and Dongfang Liu. Mixpave: Mix-prompt tuning for few-shot product attribute value extraction. In ACL, 2023.
[120] Senqiao Yang, Jiarui Wu, Jiaming Liu, Xiaoqi Li, Qizhe Zhang, Mingjie Pan, and Shang- hang Zhang. Exploring sparse visual prompt for cross-domain semantic segmentation. arXiv preprint 2r3iv:2.303.08 09799, 2023.
[121] Xue Ying. An overview of overfitting and its solutions. In Journal of Physics, volume 1 1168, pp. 022022, 2019.
[122] Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnex- plainer: Generating explanations for graph neural networks. NeurIPS, 2019.
[123] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung. Mojtaba Seyedhosscini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[124] Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint 1xXv:22111111322 2021.
[125] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Unified vision and language prompt learning. arXiv preprint arXiv=2210.07225, 2022.
16

---

Published as a conference paper at ICLR 2024
[126] Matthew D Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
[127] Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arXiv:1910.04667. 2019.
[128] Gongjie Zhang, Kaiwen Cui, Tzu- Yi Hung, and Shijian Lu. Defect-gan: High-fidelit defect synthesis for automated defect inspection. In WACV, 2021.
[129] Jeffrey O Zhang, Alexander Sax, Amir Zamir, Leonidas Guibas, and Jitendra Malik. Side- tuning: a baseline for network adaptation via additive side networks. In ECCV, 2020.
[130] Yabo Zhang, Zihao Wang, Jun Hao Liew, Jingja Huang, Manyu Zhu, Jiashi Feng, and Wang- meng Zuo. Associating spatially-consiste grouping with text-supervised semantic segmen- tation. arXiv preprint arXiv:2304.01114 2023.
[131] Haoxi Zhong, Chaojun Xiao, Cunchao Tu, Tianyang Zhang. Zhiyuan I. Liu, and Maosong Sun. How does nlp benefit legal system: A summary of legal artificial intelligence. arXiv preprint arXiv:2004.10 12158, 2020.
[132] Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, 2016.
[133] Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, et al. A comprehensive survey on pretrained foundation models: history from bert to chatgpt. arXiv preprint arXiv:2302.09419 2023.
[134] Han Zhou, Xingchen Wan, Ivan Vulić, and Anna Korhonen. Autopefi: Automatic configura- tion search for parameter-efficiet fine-tuning. arXiv preprint arXiv:2301.12132, 2023.
[135] Peipei Zhu, Xiao Wang, Lin Zhu, Zhenglong Sun, Wei-Shi Zheng, Yaowei Wang, and Chang- wen Chen. Prompt-based learning for unpaired image captioning. IEEE Transactions Multimedia, 2023.
[136] Liu Zhuang, Lin Wayne, Shi Ya, and Zhao Jun. A robustly optimized bert pre-training ap- proach with post-training. In CNCCL, 2021.
[137] Luisa Zintgraf, Taco S Cohen, Tameem Adel, and Max Welling. Visualizing deep neural network decisions: Prediction difference analysis. arXiv preprint arXiv:1002.04555, 2017.
17

---

Published as a conference paper at ICLR 2024
SUMMARY OF THE APPENDIX
This supplementary contains additional details for the twelfth International Conference on Learn- ing Representations submission, titled "Facing the Elephant in the Room: Prompt-Tuning Full finetuming? ?" The supplementary is organized as follows:
A shows comprehensive training/testing curve for VTAB-1k benchmark and cover com- prehensive experiments overfitting datasets.
B presents visualization inspections new-added explanation method Integrated Gradients, and more results on GradCAM.
SSshows the per-task results full finetuning and visual prompt tuning on different back- bone and prefraining objectives.
D
presents the image examples from the VTAB-1k image classification benchmark.
shows the optimal prompt length for each task in VTAB- Ik image classification bench-
mark.
s further presents the FID scores from a new-added image classification benchmark FGVC (37), and compare the performance between visual prompt tuning and full fine-tuning.
G is the discussion of legal/ethical considerations, reproducibility, social impact, limita- tions and future work.
Full fine-tuning
Prompi luming
Mixed
FT-then-P'
Caltech101
Pets
10
Training Loss
Testing Loss
1011 10
Training
Testing
30
10
20
Epochs
40
100
20
Epochs
40
60
160
Fpochs
40
60
60
103
60 Fpochs
60
EuroSAT
Rosise45
10-1
Testing eoss
Training Loss
Testing Loss
Training
10-
zon
10'9
40
09
150
65
Epochs
Epochs
Epochs
Epochs
Clevr/count
SmallNOORB/azmuuth
10r
10"
Training Loss
Testing Loss
Training
Testing
10-1
Epochs
Epochs
Epochs
180
Fpochs
Figure 9: Log-scale version of Figure Consistent to our paper, colors represent four
training strategies: full finetuning, prompt tuning, mixed and FT-then-PT. respectively. Same for
Figure 10
Figure
and 12
A PER-TASK TRAINING/TESTING CURVE
In Figure g further provide the log-scale version of Figure
In Figure 10 and 12 we comprehensively present per-task training/testing curve on VTAB-
1k [177 Natural, Specialized and Structured, respectively. We do not observe overfitting for full
18

---

Published as a conference paper at ICLR 2024
Full fine-tuning
Prompt tuning
Mixed
FT-che-PPT
CIFAR-100
Caltech101
Training Loss
Testing Loss
8
Training Loss
Testing Loss
20
40
60
100
50
50
120
60
00
100
20
40
60
120
Epoctts
Epochs
Epochs
Epochs
DTD
Flowers 102
Training Loss
N
Testing Loss
Training Loss
Texting Loss
20
100
60
80
60
100
Epochs
Epochs
Epochs
Epochs
Pets
SVHN
Trining Loss
Rosting Loss
Training Loss o
Testing Loss
Epochs
Epochs
Epochs
Epochs
Sun397
Training Loss 1
60
80
100
20
60
60
69
100
Epochs
Epachs
Figure 10: Per-task training/testing loss curve for VTAB-Ik 127 Natural.
finetuning in 10 19 cases, while of 19 datasets suffer from overfitting for both full finetuning and prompt tuning. Further experiments in Figure [13 on increasing training samples reduce over- fitting for both methods. Overall, it can be inferred that overfitting is not the underlying cause of performance degradation observed in full finetuning when compared to prompt tuning.
In Table and 6 we provide per-task results in accuracy with different dataset scales (See Figure 2, and $5.1 in our paper. Not all datasets are provided in the same scale due to the limit number of data samples). We further provide comprehensive accuracy curves with different dataset scales in Figure 14 15 and 16 for VTAB-1k Natural, Specialized and Structured, respectively, which covers more combinations that are not showed in Table [ The figures and Tables are consistent with our observations and support our assumption in the main paper that the performance gap between full finetuning and prompt tuning decreases as the amount of training data increases, with full finetuning even outperforming prompt tuning under high-resource settings, demonstrating the robustness and generality of prompt tuning in the face of limited finetuning examples. Though full finetuning generally achieves higher accuracy when rich data examples are available, the prompt tuning still has competitive performance with much fewer parameters.
19

---

Published as a conference paper at ICLR 2024
Full fine-tuning
Prompt tuning
Mixed
FT-the-PP
Patch Camelyon
EuroSAT
Training Loss
10-
Testing Lors
B
Training Loss
Testnn Loss
100
Epochs 80
Epochs
Epochs
Epochs
Resisc45
Retinopathy
& 8 8. x
Training Los8
Testing Loss
Training Loss
Testing
7
B 2 8
20
40
100
60
100
20
100
Epochs
Epochs
Epochs
Epochs
Figure 11: Per-task training/testing loss curve for VTAB- Ik [17] Specialized.
Full fine-tuning
Prompt tuning
Mixed
FT-then-PT
Clevr/count
Clevr/distance
Training Lass
Tesing Loss
Trining 12
Testng Loss
20
Epochs
60
80
100
20
40
Epochs
60
160
0
20
Epochs
60
00
130
20
Eguchs
40
20
165
DMLab
KITTI/distance
Training 9
Loss
Training Loss
Testing oss
20
Epochs
60
80
100
20
40
Epochs
20
100
70
00
Epochs
60
300
Epochs
60
00
160
dSprites/location
dSpritesiorientaatio
Training Loss
Teding Loos
Testing roos
Training
Epochs
Epochs
Epochs
Fpochs
SmallINOR//zimutth
SmalINORRkllevttion
Training Loss
Testng Loss
Training Loss
Tessng Lovs
Epochs
1pochs
Epochs
Fpochs
Figure 12: Per-task training/testing loss curve for VTAB-1k 127 Structured.
20

---

Published as a conference paper at ICLR 2024
Full fine-caning
Prempt
Training loos
Toeinn loss
Candlyo (1103)
Caraelya (10000)
KirTU (1003)
KITTIMbaaee (1220)
Fepech Clovoorm (500))
Tpocts Covricaare (10000)
Fpochs deprinsltentton (1000)
Tipocte
dippiiatteation (10S0e)
Logs
Ippocte Clevedistaaee (10003)
Tppcrs
Cleveliitanne (5000)
diprite Stpniisstiinniiinnss.
Epeeffic
Trpacte"
"Tipeche"
DM (10000)
SrolNsRRlaarmmh (2000)
SeattNoORiattraa.ccoo (10000))
Lips
Epecta
Epodae
Epocla SmallS clevation (10000)
Epodtt
SmailNCHIS slovation (5000)
"Ippehe"
Figure 13: Training/testing loss curve for VTAB-1k [127] when increasing training samples. Note that the listed 9 cases are observed in overfitting on both full finetuning and prompt tuning under default numbers of training samples. We further increase the number of training samples and find that both methods are mitigated from overfitting. Further enlarging in training data results in unacceptable training time and fails to cover some cases in this plot (i.e., not having enough training samples). (.) represents the number of training samples. For most cases, we apply 5000/ 1000 number of images for training. VTAB-1k [127 Structured KITTI/distance (32. the other hand, does not have sufficient data for training, we present results for 1000/1200
Table 5: Image classification accuracy on different scales on VTAB-1k [27] Natural and Spe- cialized for ViT-B/16 (24) pretrained on supervised ImageNet-21k 189 Note that not all combi- nations in Figure are listed in this table. "# Samples" represent the number of training samples during finetuning. We pick 400, 800, 5000 and 10000 number of training data as the general number of samples across all datasets. All results are averaged on five runs. Same for Table 6
(85.8M)
VIT-R166
24
Samples
CIFAR- 100 Caltech101 DTD Flowers102 Pets SVHN Sun397
VTAR-IK [27]
Natural
Mean
Patch Camelyon EurSAA Resisc45 Recinopathy
VTAR-IK [22] Specialiged (4)
FT
47
400
51.1
76.1
55.1
92.9
79.4
73.6
27.1
65.04
76.2
96.3
74.4
73.6
80.15
70.6
97.5
73.67
62.5
76.1
77.35
FT 47
800
64.5
88.1
65.0
97.3
84.6
87.3
39.5
75.19
81.2
95.7
83.2
73.3
83.48
77.7
98.1
51.0
79.53
77.9
96.2(9)
83.3
73.1
82.65
FT 47
5000
85.0
95.5
56.8
79.13
89.1
98.6
93.7
76.6
39.30
87.1
92.2
69.4
82.9
86.3
98.3
92.1
74.3
87.75
FT 47
10000
88.6
96.4
66.8
83.93
89.0
98.9
95.3
77.6
90.20
VPT
49
10000
88.7
94.0
72.6
85.10
87.7
98.6
93.9
74.6
88.70
B VISUALIZATION INSPECTIONS
We further introduce Integrated Gradients [100] (IG), and present visualization inspections in Figure
17 a) to support the advantages of visual prompts discussed in our paper. In general, IG tries to
attribute the prediction of deep network to its input features, and it is commonly applied [61 2
in the research of explainable Artificial Intelligence (AD). IG gains widespread adoption as an interpretability technique, owing to its versatility in explaining any differentiable model, including
21

---

Published as a conference paper at ICLR 2024
Table 6: Image classification accuracy on different scales on VTAB-1k 127 Structured for
VTT-B/16
24
pretrained on supervised ImageNet-21K 89].
B/T
Samples
VTAB 1k 127 Structured [8]
dSpritesloc. dSpritesorri. SmalINORB/az SmallNORRVelev, Mean
Clevrient. Clevrídist.
FT 47
400
45.3
54.3(1)
34.7
72.2
38.4
32.6
17.7
23.3
39.82
VPT
49
60.8
54.3(1)
41.2
67.0
69.0
38.4
21.5
29.4
47.70
58.2(2)
40.4
74.7
47.0
26,2
48.07
VPT 49
58.2(3)
45.3
71.7
78.4
48.4
31.2
56.09
FT 47
5000
84.4
74.1
58.4
96.1
73.1
78.9
67.95
66.62
VPT 49
5000
76.2
77.8
57.8
93.3
70.2
73.5
68.02
73.83
FT 47
10000
93.4
80.7
63.9
98.8
79.4
94.0
83.4
84,80
VPT
49
10000
84.4
81.6
63.4
96.6
76.7
87.2
80.3
81.46
100
90
Acauracy (%)
70
60
Prompt tunligg
CIFAR-100
Caltech101
40
DTD
Hower
30
SVHN
Sun397
20
103
104
Log-scale number of training set samples
Figure 14: Analysis of dataset capacity on VTAB-1k [127) Natural. The solid lines stand for full finetuning and the dotted lines represent prompt tuning. We take the log-scale of training data samples for better separation and each color stands for an individual classification task. Each point given by average five In general, with the increasing of dataset in size, the performance gap between full finetuning and prompt tuning becomes closer. Same for Figure 15 and 16
95
90
Acouracy (%)
80
75
Full fine tenettnning
Prompt tuning
Camelyon
EuroSAT
65
Hesisc45
Retinopathyy
60
10a
10a
Log-scale number of training set samples
Figure 15: Analysis of dataset capacity on VTAB-1k [127] Specialized.
22

---

Published as a conference paper at ICLR 2024
100
80
Accuracy (%)
60
Full fine tuning
Prompt tuning
Clevricnt.
Clevridist.
DMLab
KITTUdist.
20
dSprites/lac.
dSprites/ori.
SmalINORE/az:
SmallNoRBvellev.
0
10a
10*
Log- scale number of training set samples
Figure 16: Analysis of dataset capacity on VTAB-1k [27
Structured.
Input
fine-tuning
Prompt tuning
Input
fine-tuning
Prompt
tuning
Input
fino-tuning
Prompt
tuning
Prompt
Integrated Gradients
(b) GradCAM
(a) Integrated Gradients (IG) [100] visual inspection of full finetuning and prompt tuning. Note that the darker regions responds to high score for class (In light of the observation that certain images exhibit consistently negative gradient, it becomes necessary to take the absolute value order to ensure consistency and save successfully across all images within the testing set. This approach results in the emergence of darker boundaries within the resulting images). From left to right are input images after standard data augmentation, IG results for full finetuning and IG results for prompt tuning. (b) More visual inspection of full finetuning and prompt tuning using GradCAM [92]. Consistent to our paper, the red regions correspond to high score for class. From left to right are input image after standard data augmentation, GradCAM results for full finetuning and GradCAM results for prompt tuning. Figure best viewed in color.
Figure 17:
images 100 4]. text 18 84), and structured data 772 122 I.. In Figure I7 [), we observe straightforward visual explanation differences (e.g., take the Coccinella septempunctata image as example, when full finetuning fails provide high gradients to make correct decision, prompt tuning instead recognize with significantly higher gradients), showing consistency with our paper.
In Figure 17),,, present more visualization inspection results for full finetuning and prompt tuning using GradCAM [92]. Overall, we present additional visual evidence to support the notion that prompt tuning encompasses actual visual explanations throughout the training process.
23

---

Published as a conference paper at ICLR 2024
Full fine-tuning
Prompt tuning
Full fine-tuning
Prompt tuning
Input
Input
KiTTl/dst.
DMLab
Retinopathy
Figure 18: Visual inspection of full finetuning and prompt tuning in other 3 groups. For KITTI- dist. and DMLab, we present the cases where full finetuning gets inferior performance to visual prompt tuning and vice versa for Retinopathy. KITTI-dist. lies in the 2nd quadrant, DMLab lies in the 3rd quadrant and Retinopathy lies in the 4th quadrant with regard to Figure Figure best viewed in color.
Table 7: VTAB-1k [17] Natural per-task results for Swin-Base [66] pretrained supervised ImageNet-2Ik. Best results among full finetuning and prompt tuning are bold. We report the "num- ber of wins" in [-] compared to full finetuning. Same for Table to 15
Swin-Base [66]
VTAB-IK [17) Natural (7)
(86.7M)
CIFAR-100 100
Calrech101
DTD Flowers102
Pets
SVHN
Sun397
Mean
FT [47
72.2
88.0
71.2
98.3
89.5
89.4
45.0
79,10
VPT 49
79.6
90.8
78.0
99.5
91.4(3)
86.4
51.7
78.78 (6)
Tuned / Total (%)
0.13
0.13
0.07
0.13
0.06
0.70
0.48
0.28
further present visualization inspection results based on Figure In Figure 8 and 17 we pri- marily focus images with similar data distribution and task, we thus further posit visualization inspection results under the setting of other three groups in Figure 18 As seen, for KiTTI-dist. and DMLab from 2nd and 3rd quadrant respectively, visual prompt tuning presents clear dense visual idence and focuses on correct positions. On the other hand, for Retinopathy lies in the 4th quadrant, full finetuning presents more stable results, focusing cells' characteristics.
24

---

Published as a conference paper at ICLR 2024
Table VTAB-1k 127 Specialized per-task results for Swin-Base 666 pretrained on supervised
ImageNet-21k.
Swin-Base 666
VTAB-IK [127]
Specialized [4]
(86. (6.7MM
Patch Camelyon FuroSAT
Resise45 Retinopathy
Mean
FT 47
86.6
96.9
87.7
73.6
86.21
VPT [49
80.1
96.2
85.0
72.0
83.33 (0)
Tuned/ / Total (%)
0.07
0.13
0.19
0.02
0.10
Table 9: VTAB-1k 127 Structured per-task results for Swin-Base [66] pretrained supervised ImageNet-21k.
Swin-Basc 66
VTAB-IK [127 Structured [8]
(86
Clevrlent. Clevidist.
DMLab KITTIdist. dSprites/loc. dSprites/ori. SmalNNORBaa. SmalINORBlelev
Mean
47
VPT 19
67.6
39.4
30.1
613
744
50.6
25.7
23.7
$1.33
Total
0.70
0.14
0.69
0.09
0.16
0.02
C PER-TASK RESULTS ON DIFFERENT PRETRAINING OBJECTIVES.
We also report the per-task results on VTAB-Ik [17
Swin-Base 666 (i.e., Table 7 and
MAE 41 (i.e., Table 10 and I2) and MoCo v3 16 (i.e., Table 13 14 and 15 respectively. Overall, the empirical results consistently reveal the attainment of superior or competitive perfor- mance gains through prompt tuning. in comparison to full finetuning, across diverse tasks within the default train-val split. Notably, these performance gains are achieved while maintaining a substan- tially reduced number of model parameters.
D IMAGE EXAMPLES
In Figure 19 we include image examples from VTAB-Ik [127] image classification benchmark, including 19 visual tasks categorized into three groups: Natural, Specialized, and Structured.
E PROMPT LENGTH
In Table 16
17 and
18
we posit the per-task prompt length, which is consistent with the original
VPT 37 approach.
F FID ON FGVC
In Table 19 we further present the corresponding FID scores on FGVC [37] image classification benchmark. Specifically, it contains 5 Fine-Grained Visual Classification, including CUB-200- 2011 106), NABirds [033, Oxford Flowers [74), Stanford Dogs [51] and Stanford Cars [31] As higher FID score might potentially lead to relatively higher accuracy full fine-tuning, and
vice versa. This is consistent with the hypothesis claimed in $4.3
G DISCUSSION
G. ASSET LICENSE AND CONSENT
The majority of Visual Prompt Tuning (VPT) 49 is licensed under CC-BB-CC0 Portions of [49 and huggingface/rrass
are available under separate license terms:
formers are licensed under Apache-2.0 Swin- Transformer [66] and ViT-pytorch [24] are licensed
under MIT; and MoCoov3 [6] and MAE [1 are licensed under CC BY 4.0 Fréchet Inception
Distance (FID) 43 91 is licensed under Apache License 2.0
25

---

Published as a conference paper at ICLR 2024
Table 10: VTAB-1k [127] Natural per-task results for VIT-B/16 [24] pretrained on MAE [41]. Though prompt tuning shows inferior performance to full finetuning, its parameter-efficient nature makes applicable in parameter-sensitive scenes. Same for MOCO 16 applications.
VIT-B/I6 [24
VIAB-IK 1k [12 Natural [7]
Mean
(85.8M)
CIFAR-IOM 100 Caltech101
DTD
Flowers 102
Pets
SVHN
Sun397
FT [47
24.6
84.2
56.9
72.7
74.4
86.6
15.8
59.31
VPT [49
8.2
55.2
58.0
39.3
45.2
19.4
21.9
35,31(2)
Tunedd / Total (%)
0.20
0.20
0.15
0.10
0.04
0.54
0.41
0.23
Table 11:
VTAB-Ik 127
Specialized per-task results for ViT-B/16 [24
pretrained on
MAE 41
VT--B/66
VIAB-Ik [122] Specialized [4]
Mean
(85.8M)
Patch Camclyon EurSATT
Resise45 Retinopathy
FT 47
81.8
94.0
72.3
70.6
79.68
VPT 49
77.9
94,9
45.4
73.6
72.95(2)
Tuned / Total (%)
1.06
1.07
0.15
0.02
0.57
G.2
REPRODUCIBILITY
This paper implemented in Pytorch 80)
Experiments are conducted on NVIDIA A A100-80GB GPUs. For reproducibility, our full implementation shall be publicly released upon paper accep- tance.
G.3 MORE DISCUSSION ON DATASET CAPACITY
Previous works [107] demonstrate that prompt tuming in language also shows same trend that prompt tuning achieves better performance when lacking task-specific data.
Similarly in vision perspective, VPT 37 partially explore this question by claiming consistently advanced performance to full fine-tuning across training data sizes, which is proved to mistaken in our paper (see 44.4) when the dataset scale continue expand. Our research undertakes sub- sequent update of the claims made in the preceding research, rather than merely adopting them in their original form. Also, the results in [37 shown different dataset benchmark (i.e., FGVC (37].. Our paper contributes an additional piece to the puzzle addressing gaps in common image recognition benchmarks.
G.4 CHOOSING SINGLE TRAINING SAMPLE FOR ONE-SHOT LEARNING
We select one image per class for train and val, respectively. The choosing of the represen- tive image for each class under the one-shot learning scene is important (see $4.4). We want to ensure that the chosen image is not an outlier [117]. Instead, should share common features with other images from the same class. We thus apply method called iterative testing. Specifically, we experiment with different images from the same class and evaluate our model. If the model's performance falls in reasonable range, state that the chosen images representative. Also, data augmentation methods are applied, which helps to fix the variations of the image.
G.5
SOCIAL IMPACT
This work systematically investigates several hypotheses to demystify the mechanisms behind vi- sual prompt tuning's success. We carefully study whether its success provided by its enhanced resilience against overfitting, flexibility in task transfer, efficient learning on small datasets, or im- proved optimization due to additional dimensions. Through extensive experiments on 19 diverse datasets and tasks, reveal that surprisingly, overfitting not the root cause of full tuning's inferior performance. Prompt tuning demonstrates better generalization when there is limited data, while full tuning catches up tuning is still a competitive approach considering its parameter- nature. Overall, our paper suggests that researchers should have clear view of the task disparities and dataset scale of down-
surpasses prompt tuning when more data is presented. However, prompt
26

---

Published as a conference paper at ICLR 2024
Table 12: VTAB-Ik 127] Streutured per-task results for ViT-B/16 (24 pretrained MAE [41]
VIT-R/I6
VIAR.IK
Strattre
(85.8M)
Clevrient.
Clevsidist.
DMLab
KITTIdist. dSprites/loc.
dSpritesion. SmalNNORD/a. SmalINORBielev
Mcan
(7
67.0
59.8
45.2
75,3
72.5
47.5
30.2
33,0
53.82
VPT 49
39.0
40.9
30.6
53,9
21.0
12.1
11.0
14.88
27.9100)
Tuned/ / Tota (%)
0.54
2.11
1,07
0.54
0.12
0.55
2.12
2.11
1.14
Table 13: VTAB-IK 1k [17 Natural per-task results for ViT-B/16 [4] pretrained on MOCO [6
VIT-B/16 [24]
VTAB-IK [127] Natural [7]
Mcan
(85.8M)
11FAR-100
Caltcch101
DTD
#lowers122
Pets
SVHN
Sun397
FT [47]
91.0
64.6
91.6
79.9
89.8
29.1
71.95
VPT 49
70.1
88.3
65.9
88.4
57.8
35.7
70.26(4)
Tuned/ Total (%)
0.20
0.20
0.15
0.10
0.04
0.54
0.41
0.23
stream finetuning, and carefully select appropriate way for finetuning under "pretrain-ther thee-finnttune paradigm.
G.6 LIMITATIONS AND FUTURE WORK
Although we investigate several interesting and compelling doubts between full finetuning and visual prompt tuning, which of great significance the research community, it also comes with new challenges and unveils some intriguing questions. For example, the examination of visual prompt tuning reveals similarities to prompt tuning approaches in NLP [33 [07] and vision from different perspectives (e.g., [14 discusses that prompt tuning is highly robust to label noises; [76] demon- strates how prompt-tuning enables the model to attend to context-relevant information), thereby suggesting the need for further investigation to facilitate comprehensive and unified study. This topic can extended to parameter-efficien methods other than prompt tuning techniques. For ex- ample, we build limited experiments on LoRa [6),, showing that there is similar trend on dataset capacity when comparing with full fine-tuning (see Table 20p. Another essential future direction deserving of further investigation is the the design and analysis of network's attention position and ad-hoc interpretability.. In our paper, we follow common practice and propose network interpretabil- ity through various visualization explanation methods (i.e., GradCAM, IG) at the final layer of the network. However, we highlight that visualizing the attention from cls token to other tokens [9] be alternative approach to understand the effect of VPT on feature. We share include dif- ferent visualization position in future research. We further highlight (94] very important work in discussing the relation between attention visual evidence and performance. During our cur- explanation for the performance gain, while some current works [94] pave path for such the connection. Also, the discussed visualization approaches can be generally categorized into the post-hoc explanability
rent research, the visual evidence can hardly provide an intuitivestraightforval
(i.e., producing explanations for trained networks by importance values [97] [26 132, 27 77, 995
sensitivities of inputs [67 52 (37).. Such approaches, however, might suffer from possible 57 21. Therefore, is worth further investigating the ad-hoc explanability
misleading (87 88
of full finetuning and prompt tuning [8I] [09 (i.e., case/concept-based reasoning), particularly in
decision-critice and safety- scenarios 78] 25. The investigation prompt tuning's 's ad-hoc explanability in research is still relatively scarce and necessitates further exploration.
27

---

Published as a conference paper at ICLR 2024
Table 14:
VTAB-Ik 127
Specialized per-task results for VIT-B/16 24 pretrained on
MOCO [16].
VIT-B/16 24
VTAB-IK [127] Specialized [4]
Mean
(85.8 855
Patch Camelyon EuroSAT
Resisc45 Retinopathy
[47
85.1
96.4
74.3
84.72
83.1
91.0
81.2
74.0
82.33(0)
Tuned Total (5)
1.06
1.07
0.15
0.02
0.57
Table 15:
VTAB-1k 127
Structured per-task results for ViT-B/16 24
pretrained
MOCO [16].
VIT-B/I6 24
VTAB-IK k [12) Struetured [8]
(85.8M)
Clevrient.
Clevidiit. DMLab
dSprites/loe. dSprites/ori. SmalNNORR/a. SmallNORBlelev.
Mean
47
55.2
56,9
44.6
77,9
63.8
49.0
31.5
36.9
31.88
VPT 49
48.5
35.8
37.2
61.6
32.3
2633
19.4
34.8
Total (%)
1.07
0.54
0.12
2.12
1.14
Table 16: VTAB-1k [127] Natural per-task prompt length for ViT-Base 24 vised ImageNet-21k..
pretrained on super-
ViT-B/I6 24
VTAB-IK Ik [127] Natural (7)
Mean
(85.8M)
CIFAR-100 Caltech101
DTD Flowers102
Pets SVHN Sun397
VPT [49
200
50
200
79.58
Table 17: VTAB-IK [127 Specialized per-task prompt length for ViT-Base 24 pretrained on
supervised ImageNet-2Ik.
V/T- 24
VTAB-IK [127] Specialized [4]
Mean
(85. 8M)
Patch Camelyon EuroSAT Resisc45 Retinopathy
VPT 49
50
50
10
28.75
Table 18: VTAB-1k 127 Structured per-task prompt length for ViT-Base 666 pretrained on supervised 2etgeNet-21k.
V/1
VTAB-IK [22) Structirnd [8]
(85 MM
Clevr/ent. Clevndiss, DMLab KITTl/dist. dSprites/loc. dSpritesfori. SmalINORB/az. SmallNORB/elev,
Mean
VPT [9
100
200
100
100
100
200
200
137.5
Table 19: FGVC 49
per-task results for ViT-Base/16
24
pretrained supervised ImageNet-
21k.
VTT-Base/16 24
FGVC (49 [S]
Mean
(85.8M)
CUB-200-2011
NAbirds
Oxford Flowers Stanford Dogs
Stanford Cars
FULL
87.3
82.7
89.4
84.3
88.54
VPT
49
88.5
84.2
99.0
90.2
83.6
89.11 (4)
Tuned/ / Total (%)
0.29
1.02
0.14
1.17
2.27
0.98
FID scnre
146.845
169.117
167.690
143.800
182.487
Table 20: Cifar-100 100 (54) for VIT-Base
24
(bottom-up) pretrained on supervised ImageNet-21k
under the settings of 94
# training sample
400
800
10000
Full fine-tuning
44.5%
70.2%
87.9%
LoRa [46]
69.6%
83.6%
90.7%
Performance gap
25.1%
13.4%
2.8%
28

---

Published as a conference paper at ICLR 2024
VTAB-Ik B-lk Natural
C-FAR-100
Caltech101
DTD
19
6
58
Flowers 02
Pets
SVHN
Sun397
VTAB Specializod
Patch Camelyon
TuroSA
Resiso45
Retinopathy
VTAB-1k Strudbured
Clevifcount
Clevr/distance
DMLab
dditttiittnce
dSprtes/location
dSpriioslorientation
SmallNORR/eeimuth
SmallINORB/alevation
Figure 19: Dataset examples from VTAB-Ik [127] image classification benchmark.
29


Reviews:
[{"review_id": "0FIg2p6NKT", "reviewer": "Reviewer_2Y28", "summary": "The paper provides a deep analysis of visual prompt tuning (VPT), a popular transfer learning method for vision tasks. Specifically, the paper  discusses when VPT is more favorable than fully fine-tuning and give some insights on why VPT has higher performance than fully fine-tuning. The paper finds that when the data disparity between pre-training and downstream tasks is small, or the task disparity is large, VPT normally has higher performance. The paper demonstrates that when task disparity is large, VPT is usually better because fully fine-tuning is more prone to over-fitting, and when data disparity is small, VPT is usually better because it preserves more information from pre-training by freezing the parameters. The paper provides insights and guidance on when and why to choose VPT over fully fine-tuning.", "questions": "See Weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "The paper discusses an interesting and important subject, and gives sensible and valuable insights on the topic. The whole analysis is thorough enough and the conclusions seem credible. The paper is overall well written and easy to follow.", "weaknesses": "1. In Section 5.4, the authors show that the GradCAM maps of VPT is more interpretable and more focused on task-relevant regions than fully fine-tuning. It seems the examples are all from natural image classification. It would be helpful to compare the attention on four quadrants of transfer learning to see the difference when data/task disparity varies and how it is connected to the conclusion in Section 4. \n\n2. The visualization of attention in Section 5.4 is based on GradCAM. Have the authors tried visualizing the attention from cls token to other tokens, as in previous work in the ViT literature (e.g., [1])? On the other hand, a previous paper [2] also discusses the relation between transfer learning and attention. They find that current transfer learning methods tend to have noisy attention and fixing the attention can boost the transfer learning performance. It would be helpful to add some discussion on the relation between this work and [2].\n\n3. This paper is focused on VPT. Since other transfer learning methods such as LoRA [3] may have more favorable performances on various transfer learning tasks like VTAB-1k, readers may wonder if these methods also have similar properties. It would be helpful to see if the same conclusions also hold for other transfer learning methods. \n\n[1] Caron, Mathilde, et al. \"Emerging properties in self-supervised vision transformers.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n[2] Baifeng Shi, Siyu Gai, Trevor Darrell, and Xin Wang. Refocusing is key to transfer learning. arXiv preprint arXiv:2305.15542, 2023\n[3] Hu, Edward J., et al. \"Lora: Low-rank adaptation of large language models.\" arXiv preprint arXiv:2106.09685 (2021).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CU0dzge4zy", "reviewer": "Reviewer_8arZ", "summary": "Visual prompt tuning (VPT) and full finetuning (FT) are two commonly-used techniques to adapt a pretrained vision transformer to downstream image recognition tasks. Through extensive experiments, this paper is aiming to investigate when (under which conditions) and why VPT would outperform FT:\n1. For the when part, the authors find VPT is favored when: \n- there’s a disparity between pre-trained task and target downstream task; \n- target downstream task share similar data distribution with the pre-trained data; \n- when target labeled data is limited.\n2. For the why part, the authors give several hypotheses and experiments aiming to verify them including: \n- both VPT and FT suffer from overfitting so VPT’s success cannot be solely attributed to less overfitting during finetuning;\n- VPT’s special way of preserving original weights and adding new parameters might be the key.", "questions": "It would be great if the authors could respond to the weakness points mentioned above. Thanks!", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written with clear high-level motivation/intuition/ideas, along with detailed subsections that introduce the low-level implementation details. It is easy to follow and understand for the general audience in the field of recognition, vision transformer or transfer learning.\n2. Indeed as in the title, the problem that the paper is trying to tackle is quite important but also not fully explored yet, i.e., the elephant in the room. VPT is widely used in multiple areas as a common tool to adapt pre-trained models to downstream tasks, not only in recognition, but also in dense prediction, vision language model and even generation nowadays. However, there are very few works talking about this classic “when and why” problem such that people could understand VPT better and also use it in a better practice.\n3. Some of the extensive experiments are quite convincing, especially the analysis on downstream data scale (Sec. 4.4) and hypothesis on overfitting (Sec. 5.1).\n4. The hypothesis on (a) how data distribution and task similarity gonna affect VPT’s performance (Sec 4.3); (b) why VPT outperforms FT from an optimization perspective (Sec 5.2) are quite interesting. The experiments alongside are also very inspiring.", "weaknesses": "1. Although the hypotheses in Sec. 4.3 and Sec. 5.2 look very interesting and inspiring (as mentioned in the Strength point 4), I’m concerned the experiments are not convincing enough to support them:\n- In Sec. 4.3, it’s a bit risky to draw the conclusion using only 19 data points. Adding more tasks including the detention or dense prediction ones could be better. \n- In Sec. 5.2, although Fig. 7 is very intriguing, the experiment results are not able to support it, i.e., Mixed and FT-then-PT still fall behind VPT with clear margins. This also leads to a very ambiguous conclusion in the last paragraph in Sec. 5.2.\n2. The comparison of GradCAM visualization in Sec. 5.4 is not very informative (the distinction is not that huge, plus more like a calibration difference?) and it’s also hard to draw any convincing conclusion from it.\n3. In Tab. 1, Mixed and FT-then-PT are included without any introduction in the Sec. 4.1 alongside, which could be very confusing to the audience. Maybe remove them from Tab. 1 and make another comparison table in Sec. 5.2, where these two methods are actually introduced?\n4. The number of wins notation was introduced in Tab. 3 but has already appeared in Tab. 1 and 2 before, which could confuse the audience. Also, ‘[]’ was used for both this notation and also reference. Maybe change it to ‘()’ to avoid any confusion?\n5. A similar observation on data scale (Sec. 4.4) is actually also covered in the original VPT paper/experiments (Page 8, Fig. 3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fxIkWlOajM", "reviewer": "Reviewer_jEi5", "summary": "The authors attempt to answer the question: When and why VPT is effective. They conduct extensive experiments on 19 diverse datasets and tasks, wherein VPT outperformed FT in 16 instances (VTAB-1k [110] image classification benchmark). The model is ViT (and the authors observed the same tendency when they use Swin).\n\nIn exploring the “when” dimension, they find that VPT is preferrable when there is 1) a substantial disparity between the original and the downstream task objectives (e.g., transitioning from classification to counting), or 2) a similarity in data distributions between the two tasks (e.g., both involve natural images).\nIn exploring the “why” dimension, they find that VPT’s success cannot be attributed solely to overfitting and optimization considerations and that VPT preserves features and add parameters in a unique manner.\n\nSpecifically, \n- VPT reaches superior performance than FT in two different scenarios: (1) when the disparity between the task objectives of the original and downstream tasks is high, or (2) when the data distributions are similar.\n- The performance gap between full finetuning and prompt tuning decreases as the amount of training data increases. \n- Although full finetuning generally achieves higher accuracy when rich data examples are available, the prompt tuning still reaches a competitive performance with much fewer parameters.\n- In one-shot classification, prompt tuning outperforms full finetuning in many tasks and achieving substantially higher accuracy in some cases (i.e., 41.66% vs 56.94% in VTAB-1k Natural).\n- Training and testing losses consistently decrease in scenarios where the task objectives are similar between the original and downstream tasks. In such cases, overfitting is not the cause of performance degradation of FT.\n- The presence of additional dimensions does not significantly aid the optimization process.\n- Preserving the original feature proves to be crucial for transferring to downstream tasks, as evidenced by the superior performance of VPT compared to FT-then-PT.\n- Maintaining a fixed feature space in VPT may compel it to learn more valuable embeddings with the trainable prompts, as it significantly outperforms the Mixed method.\n- Therefore, pretrained parameters play a pivotal role in capturing general features in transfer learning. That is, the preservation of initial\nfeatures is important for VPT’s success, but in a very sophisticated manner.\n\nOverall, the authors find many characteristics of VPT in the context of \"FT vs. VPT,\" although some of them might be an overstatement.", "questions": "- [Question] The experiment is limited to Transformer-based models. Do you think all the results are also valid when we use CNNs?\n\n- [Comment (major)] In Figure 4,\n> In general, with the dataset increasing in size, the performance gap between FT and VPT becomes narrow.\n\nI would like to see more results before the conclusion because Figure 4 shows that some of curves are not monotonic, which is counterintuitive, and that the behaviors of the curves highly depends on the data and methods.\n\n- [Question (major)] Figure 4: Does this result include the randomness introduced by the choice of the training set samples? The broken yellow curve in the left panel looks intensely fluctuating in the small set region, and I guess this fluctuation comes from that kind of randomness.\n\n- [Question (major)] In one-shot classification experiment (Table 3), how did the authors choose the single training sample? The performance should highly correlate with the one-shot training sample.\n\n- [Comment] At the end of Section 4,\n> These results further support our assumption that prompt tuning is more effective than full finetuning when only limited finetuning examples are available.\n\nDoesn't this statement conflict with the Patch Camelyon result (green curves) in Figure 4?\n\n- [Comment] Figure 5 should use log scales. Also, all figures including Figure 5 should be vector images.\n\n- [Comment] In Section 5.2,\n> Another possible explanation for why prompt tuning can achieve superior performance is that it can escape from local minima or saddle points compared to full finetuning, due to the additional dimensions introduced by the prompts.\n\nThis is clearly an overstatement (or a misuse of words). Local minima and saddle points are rigorously defined in a mathematical way. I guess what the authors mean here is if we can observe performance gain or not. However, strictly speaking, escaping form local minima or saddle points is not  a necessary nor sufficient condition of generalization in general settings. Although I understand some papers use these words interchangeably, and my comment may be a bit nitpicking, I would like to recommend that the authors modify the statement here. By the way, it will be a dramatic research if one theoretically proves that VPT in fact makes the network escape from local minima or saddle points.\n\n- [Comment] In Section 5.3, the authors make two hypotheses. Is there any idea to validate them in an objective and qualitative manner?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Topic is timely, challenging, and important in machine learning community.\n\n- The experiments are well-organized. The results are clearly shown and easy to follow.\n\n- Error bars are provided when necessary.\n\n- Full implementation will be publicly released for reproducibility.\n\n- The present paper is well-written and easy to follow. Logic flow is smooth.\n\n- Overall, the authors find many characteristics of VPT in the context of \"FT vs. VPT,\" which contribute to the community.", "weaknesses": "- [Minor] The experiment is limited to Transformer-based models.\n\n- There may be some overstatements, e.g., in Figure 4 (see Questions).\n\n- The reason why VPT outperforms FT in scenarios characterized by distinct data distributions and high task disparity is still unclear, although the authors make a hypothesis (Section 5.3).\n\n- The reason why FT outperforms VPT in situations involving similar tasks with varying data distributions is still unclear, although the authors make a hypothesis (Section 5.3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Lph5QOLVBW", "reviewer": "Reviewer_5V4Y", "summary": "This paper mainly focuses on visual prompt tuning. The authors conduct extensive empirical study to illustrate when and why VPT performs better than vanilla finetuning. The experiment results lead to interesting conclusion with regard to the discrepancy of task objectives and data distributions. The authors further inspect the role of solving overfitting and optimization in two kinds of methods.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The experiment results are sufficient.\n2. The idea of explaining VPT with the different role of additional prompt parameters and original pretrained parameters is interesting.", "weaknesses": "1. I suggest the authors reorganize the experiment results. The current ones are confusing since the Mixed and FT-then-PT are introduced in the later part of the paper while the corresponding results are shown before.\n2. I wonder if prompt size is one of the tuned hyper-parameters. If so, it would be better to present the exact number of prompts used for each setting since different prompt size indicates different learning capacity.\n3. In fact the original VPT has two variants, i.e. VPT-shallow and VPT-deep. It seems VPT-deep is adopted in this paper. I wonder if the same pattern holds for VPT-shallow too.\n4. It is noteworthy that VPT only performs better in half of all datasets of VTAB-1k Structured in one-shot setting as shown in Tab.4, which is different from results of other two sub-categories. This should be highlighted and illustrated in Sec.4.4.\n5. The current scope of this paper is ok. But it would be much greater if the authors could expand the study to prompt tuning in language models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)