Unleashing the Power of Visual Prompting At the Pixel Level
Authors: Junyang Wu
Xianhang Li
Chen Wei
Huiyu Wang
Alan Yuille
Yuyin Zhou
Cihang Xie
Venue: Accepted by TMLR ()

Abstract: This paper presents a simple and effective visual prompting method for adapting pre-trained models to downstream recognition tasks. Our approach is underpinned by two key designs. First, rather than directly adding together the prompt and the image, we treat the prompt as an extra and independent learnable entity. We show that the strategy of reconciling the prompt and the image matters, and find that warping the prompt around a properly shrinked image empirically works the best. Second, we re-introduce two “old tricks” commonly used in building transferable adversarial examples, i.e., input diversity and gradient normalization, into the realm of visual prompting. These techniques improve optimization and enable the prompt to generalize better. We provide extensive experimental results to demonstrate the effectiveness of our method. Using a CLIP model, our prompting method registers a new record of 82.5% average accuracy across 12 popular classification datasets, substantially surpassing the prior art by +5.2%. It is worth noting that such performance not only surpasses linear probing by +2.2%, but, in certain datasets, is on par with the results from fully fine-tuning. Additionally, our prompting method shows competitive performance across different data scales and against distribution shifts.

Published in Transactions on Machine Learning Research (05/2024)
Unleashing the Power of Visual Prompting At the Pixel Level
Junyang Wu
SJTUwjy edu.cr
Shanghai Jiao Tong University
Xianhang Li
zionhang'10@@mmallcom
UC Santa Cruz
Chen Wei Johns Hopkins University
weichenn30112@mmail.com
Huiyu Wang
williammuenghaiiqu@@maaa..ccm
FAIR, Meta
Alan Yuille Johns Hopkins University
uguille1@@hue edu
Yuyin Zhou UC Santa Cruz
shoupuginer&gmadlccco com
Cihang Xie UC Santa Cruz
cirie@ucsc. edu
Reviewed on OpenReview; https: // openreview. net/ forum? id- YgutLs4ri
Abstract
This paper presents simple and effective visual prompting method for adapting pre-trained models to downstream recognition tasks. Our approach is underpinned by two key designs. First, rather than directly adding together the prompt and the image, we treat the prompt as an extra and independent learnable entity. We show that. the stralegy reconciling the prompt and the image matters, and find that warping the prompt around properly shrinked image empirically works the best. Second, we re-introduce two "old tricks" commonly used in building transferable adversarial examples, f.e., input diversity and gradient normalization, into the realm of visual prompting. These techniques improve optimization and enable the prompt to generalize better. We provide extensive experimental results to demonstrate the effectiveness of our method. Using a CLIP model, our prompting method registers a new record of 82.5% average accuracy across 12 popular classification datasets, substantially surpassing the prior art by +5.2%. It is worth noting that such performance surpasses linear probing by +2.2%, but, in certain dalasets, is on par with the results from fully fine-tuning. Additionally, our prompting method shows competitive performance across different data scales and against distribution shifts.
only
1 Introduction
Deep learning models have witnessed pre-training increasingly large-scale datasets general and ef- fective pathway to succeed in both computer vision (He et al. 2022 Radford et al. 2021 Bao et al.) 2021 and natural language processing (Devlin et al. 2018 Brown el al. 2020 Liu et al. 2019). These pre-trained models are termed foundation models (Bommasani et al. 2021). While fully fine-tuning stands as one the most prevalent paradigms to effectively adapt these foundation models range of downstream tasks, it can be computationally intensive due the large number of training parameters. This has led a need for more efficient alternatives for adapting these cumbersome foundation models to new tasks.

---

Published in Transactions on Machine Learning Research (05/2024)
89
88
87
-2500x less parameters
88.1
86
85
Top-1 Averaged Accuracy
84
68
82.5
EVP (Ours)
80
80.3
Zero-shot
R
78
VP
77
77.3
VPT
76.5
76
probing
51.8
Fully fine-tuning
51
10a
10c
10
10
The number of parameters (log-space)
Figure 1: The trade-off between the number of parameters and the average accuracy (across 12 datascts). Our method outperforms linear probing and other visual prompting baselines by significant margin with similar number of parameters.
Prompting method, which only modifies the input space, offers an effective and efficient solution in NLP Gao
et al. 2021 Lester al. 2021 Li & Liang] 2021), e-g., text prompting can closely match the performance of fully fine-tuning (Liu et al., 2021b). This promising result has motivated researchers probe whether similar success can be achieved in the field of computer vision. Some early efforts in this direction include VPT (Jia et atal. 2022) and VP (Bahng et al. 2022), which add small amounts learnable parameters as tokens or perturbations directly at the pixel level to adapt foundation models. However, when taking a closer look at the trade-off between performance and parameter efficiency as shown in Figure [ we note these advanced visual prompting methods appear be less competitive. To illustrate, despite having comparable number of parameters, there exists a significant, performance gap between VPT and the simple linear probing baseline (77.3% us. 80.3%). In this paper, aim to unleash the full potential of visual prompting at the pizel level, and, more importanlly, lo erplore whether il can be stronger than other allernatives such linear probing.
Intriguingly, we find that with proper modifications, visual prompting can evolve into a truly effective paradigm for adapting foundation models to different visual tasks. The first key observation is that directly adding logether the prompt and the image; as in VP (Bahng al. 2022), may inadvertently distort the intrinsic information of the image, thereby limiting the prompt's learning potential. We provide a simple stratogy to resolve this issuc: first shrink the original image into smaller size and then pad the prompt around it, f.e., the prompt and the image allows for independent optimization of the prompt and enables flexibility in adjusting the padding size to control computation overheads. In addition, we draw on techniques from adversarial examples, which have similarities to visual prompting in their aim to either maximize or minimize the loss function (Bahng et al. 2022 Flsayed et al. 2018 to further improve the performance of visual prompting. Specifically, we find that gradient normalization Goodfellow al. 2015 Dong et al. 2018)) and input diversity (Xic et al. 2019) are elfective at improving the generalization ability of the prompt.
kept separate entities, free from overlap. This strategy
We follow the standard evaluation protocol to conduct experiments across 12 visual benchmarks. We note that, with CLIP model (Radford et al. 2021), our method attains an averaged accuracy of 82.5%, sig- nificantly outperforming the previous stale-of-the-art visual prompting method VPT (Jia et al.] (022 by +5.2%. More excitingly, this 82.5% result is +2.2% stronger than the linear probing result (80.3%) and even comparable to fully fine-tuning on certain datascts. We further confirm the superiority of our method in learning with data different scales and in handling out-of-distribution samples. We hope our study can catalyze further research in the field of visual prompt. learning.
2

---

Published in Transactions on Machine Learning Research (05/2024)
2 Related Works
Prompt lcarning in NLP.
The key idea of prompting is to reformulate the input text in downstream lasks so that the frozen language models can better "undersland" and perform the task (Liu et al. 2021a)
Prior works (Brown et al. 2020 Petroni et al. 2019 Cui et al., 2021 show that manually designed text prompt can help language models achieve remarkable representation capacity in the few-shot or even zero- shot settings at downstream tasks, but this requires specific domain knowledge. To address this issue, recent works have started to focus on prompt tuning (Li & Liang 2021 Liu et al. 2021b Lester et al., 202T which involves directly optimizing the continuous prompting vector through gradient information. In this work, we investigate prompt learning in computer vision, which is more challenging task because it involves a different type of signal (visual rather than language) that contains much less high-level semantic information.
Visual prompt learning. After witnessing the success of prompting in language models, researchers begin to explore the usage of prompts the field of computer vision. As a pioneering work, visual prompting is applied as strategy of model programming Tsai al. (2020). For example, CoOp (Zhou al.) 2022) applies prompt tuning to vision-language models, lcarning the soft prompts through minimizing the classification loss on downstream lasks. VP (Bahng et. al.) 2022) and VPT (Jia et al.) 2022) focus on prompting with images: VP optimizes the prompt directly in the pixel space, and VPT proposes to insert a set of learnable tokens into VIT architectures (Dosovitskiy et al., 2020) for prompt tuning, In addition, 1LM Chen et al. (2023) and AutoVP Tsao et al. (2024) explore visual prompting from broader perspectives. ILM proposes a label mapping strategy to alleviate the issue of mapping precision. On the other hand, AutoVP considers various aspects such as imput scale, visual prompts, pre-trained model selection, and output LM strategies to design high-performanc prompt methods automatically. While these approaches show the potential of visual-only prompt learning, shown in Figure [ their performance is not as competitive compared to other methods such as linear probing. In this work, we aim to enhance visual prompt learning and demonstrate its strong potential for improving the performance of foundation models on a range of visual tasks.
Adversarial examples. It is well-known that machine learning models are vulnerable to adversarial at- tacks (Dalvi al.] 2004 Biggio et al. 2013 Huang et al., 2011). The fast gradient sign method (FGSM) Goodlellow et al. 2015) and projected gradient descent (PGD) (Madry et al. 2018) are two commonly used techniques for creating adversarial examples that can fool deep learning models. Nonetheless, these adver- sarial examples cannot transfer well to fool other models. Later works show that the difficulty of optimizing adversarial examples is the cause of this weak transferability, and techniques such as diverse input patterns (Xie et al. 2019 and momentum-based gradient accumulation (Dong et al., 2017 have been proposed to improve transferability. Given the similarity between generating adversarial examples and the process of prompt learning (Bahng et al. 2022 Elsayed et al.) 2018), we hereby are interested in revisiting techniques from building transferable adversarial examples to enhance visual prompting.
3 Methodology
In this section, we present Enhanced Visual Prompting (FVP), simple and effective pixel-level visual prompting method for adapting foundation models to downstream tasks. We first provide thorough review of previous visual prompting methods as preliminaries, including VP and VPT, and then describe the prompting design and training strategy of our EVP in detail.
3.1 Preliminaries
VPT (Jia et al. 2022) adds set of learnable parameters into ViT architecture for visual prompting. For a fair comparison, we hereby consider its VPT-SHALLOW version, which only inserts the prompt into the first layer's input. Specifically, as shown in Figure 2 (a), VPT inserts a collection of prompts P between the learnable class token [CLS] and a sequence of patch embeddings E, creating a new input x = [CLS,, P..
VP Bahnn et al. 2022) adapts foundation models to downstream tasks by directly adding together the learnable prompt and the input images at the pixel level. The prompt Us is designed to be input-agnostic
3

---

Published in Transactions on Machine Learning Research (05/2024)
Resize
Visual Prompt
Normalize
Augment
Embed Layer
Visual Prompt
Image patch
Visual Prompt
embeddings
CLS
Transformer E Encoder Layer
Image Encoder
Transformer Encoder Layer
Image Encoder
Image feature
Image feature
Image feature
(a). VPT
(b). VP
(c). Ours
Figure 2: Overview of the designs of different visual prompting methods. (a) VPT: Injecting set of learnable
parameters inta the token space; (b) VP: Modifying learnahle perturhations the border of input images; (c) Our EVP: Shrinking images and applying data augmentations, then padding the prompt around the image. Note the prompts in EVP updated using normalization strategies inspired adversarial attack techniques (Goodfellow et 2015
and lask-specific, and is placed on the border of the input images, as shown in Figure 2 (b). During training, VP maximizes the likelihood of the correct label y by optimizing the prompt ro: max,, P(ylæ + n2). During inference, the optimized prompt is then added test images: Xtest {stest + Udr..., Flest + Uo).
3.2 Designing EVP
Our prompting design is largely based on VP, but with some simple modifications. The issue we identify with VP that directly adding together the prompt and the images may corrupt the original image information. For example, Figure (b), the cat ears are heavily overlapped and obscured by the added prompts. This could hinder the learning of prompts (see our ablations Section 5.1. To address this issue, as shown in Figure 2 (c), our EVP shrinks the input images and pads the prompt around them. Specifically, for an input image X E Kxxx3 it is shrunk to å kxx33 and then padded with (K² k2) 3 prompts to obtain the output image X RRRKX Similar to VP, during training we optimize the prompt by maximizing the likelihood of the correct label, and during inference we pad the optimized prompt around the shrunk test samples for predictions.
It important to note that while both EVP and VPT keep the prompt and the image non-overlapping, there is a key difference between these two methods. Specifically, the prompts in EVP are later addod with positional embedding, while this not the for VPT. As shown in the ablation study in Section 5.2 positional information is crucial for achieving strong performance with visual prompting.
3.3 Training Strategy of EVP
There is a strong relationship between prompting and adversarial attacks. In adversarial attacks, the goal is Iearn pixel perturbation 9; that will mislcad the network given an image 2, This can be formulated ming. P(y Jr, + gt). On the other hand, visual prompting can be seen as the inverse process of adversarial attacks, in which the aim is to learn template that will maximize the likelihood of the correct label y.
4

---

Published in Transactions on Machine Learning Research (05/2024)
Given this relationship, we motivated to explore whether techniques from adversarial attacks, particularly those focused on building transferable adversarial examples, can be useful for visual prompting.
Input diversity. Previous work (Xie el al. 2019) demonstrates that inpul diversity can help optimization and improve the transferability of adversarial examples. As shown in later works, the concept of "diverse input" can be gencralizod to apply different data augmentation strategies in generating adversarial examples (Dong et al. 2019 Wang et al.
2021 Wu et al., 2021). We hereby re-introduce this concept into visual prompting. Specifically, our EVP considers a range of augmentation including RandomHorizzotallElinn Ran- domCop, RandAug (Cubuk et al.
2020), and Cutmix (Yun al. 2019 As shown in Section 5.3 we find that simple augmentation strategies like RandomHorizontallFlip and RandomCrop are already sufficient to significantly improve visual prompling.
Gradient normalization. In adversarial attacks, it is common to apply normalization techniques, such as the L1, Lg, or Loe norm, to the gradients update (Goodfellow et al. 2015). For example, using the La norm, the gradient can be normalized as follows:
Vx,(1,y) gado I +7 [V,
(1)
where y the learning rate, is the loss function, and V.J is the gradient of the loss function w.r.t. the input z.
We hereby introduce this gradient normalization to visual prompting. We define the matrix representation of EVP V. = W o M, where W E xx3 are the prompt parameters, M E R Kx3 xx is the mask matrix, and @ denotes the element-wise matrix product. The mask matrix M encodes the spatial locations of the prompls, with the central part of size k x l being all zeros and the rest being all ones. In practice we find that dividing the gradient of EVP by the L2 norm the gradient of W leads to the best performance:
V
V V
(2)
We provide more details and ablation results on different normalization strategies in Section 5.3
4
Experiments
Datasets. We evaluate visual prompting methods on 12 downstream classification datasets, including CIFAR100, CIFAR10 Krizhevsky et al. 2009), Flowers102 (Nilsback Zisserman 2008), Food101 (Bossard (Xiao et al.) 2010), SVHN (Notzer et al. 2011). DTD (Cimpoi et al. 2014), OxfordPets (Parkhi et al. 2012), Resisc45 (Cheng et al. 2017), CLEVR (Johnson
ct al. 2014). EuroSAT (Helber al.J 20199) SUN397
test the robustness of visual prompting 3 oft-oo-ddittrbution datascts (Koh et al. 2021 (Camelyon17, FMoW, and iWildCAM), and 2 corruption
et al. 2017), and DMLab (Beattie et al.]
2016). In addition,
datasets
[Hendryeks & Dietterich 2018)
CIFAR100-C and CIFAR10-C).
We compare the performance of EVP with other commonly used prompting methods and fine- tuning protocols, including TP (text prompting), VP, VPT, LP (linear probing), and FT (fully fine-tuning). Specifically, we should note 1) TP is equivalent to zero-shot in CLIP; 2) LP uses a linear layer as the classification head; and 3) FT updates all paramcters of the backbone and the classification head.
Baselines.
4.1 CLIP
Following the protocol of VP (Balng et al. 2022), we conduct evaluations using the CLIP-Base/33 model on 12 classification datasets. The full results are shown in Table and a detailed comparison to the two strong basclines, LP and VPT, presented Figure 3. Our proposed EVP approach consistently outperforms all previous prompting methods, with similar or fewer parameters. On average, EVP shows an improvement of 6.0% over VP, and 5.4% over VPT.
5

---

Published in Transactions on Machine Learning Research (05/2024)
Table 1: Performance comparison across 12 datasets with CLIP. EVP substantially heats other visual prompting methads by large margin. More notably, EVP outperforms the linear probing on 7 out 12 datasets with similar number of parameters. The results where EVP outperforms linear probing highlight in bold.
Adapiation
Tinable paraue (M)
CIFAR10U
CIFARI0
Flowers
Food
EuroSAT
SUN
DMLab
SVHN
Pele
DTD
RESISC
CLEVR
Avg.
TP
0
63.1
89.0
61.8
55.2
31.1
58.0
30.22
11.0
$5.9
42.5
12.4
20.2
51.8
VP
0.070
73.3
94.2
02.0
83.2
95.6
68.4
11.9
83.4
86.5
37.1
51.1
81.4
76.5
VPT
0.061
76.6
95.0
76.2
81.7
91.6
69.3
18.1
86.1
61.6
513
58.6
77.3
EVP(Ouus)
0.062
81.2
96.6
82.3
84.1
97.6
71.0
62.3
90.5
90.0
08.4
59.7
75.9
80.5
LP
0.037
$0.0
95.0
91.1
88.3
91.8
76.2
19.3
65.1
89.2
73.5
92.3
66.1
80.3
151.28
82.1
85.8
97.4
87.8
99.0
79.0
63.5
95.7
88.5
72.3
98.1
94.4
88.1
SVHN
+25.1
CLEVR
+17.3
DMLab
+13.0
DMLab
+13.99
CLEVR
+9.8
RESI5C45
EuroSAT
+2.8
DTD
+6.1
CIFARI0
Flowers102
+1.2
CIFAR100
+4.6
OxfordillPPet et
+0.8
SVHN
+4.4
EuroSAT
+3.0
-2.6
RESISC45
SUN
+1.7
.4.
Food101
CIFAR10
-5.1
DTD
-5.2
SUN
-0.6
Food101 OxfordPets
-1.88
-2.1
-12
110
-8
-6
-4
-2
0
2
4 6
10
12
14
16
18
20
22
24
26
15 16 17
P Performanc gain(%)
P Performance gain(%)
(a). EVP vs Linear Probe
(b). EVP vs
Figure Performance gain EVP compared to linear probing and VPT on each downstream dataset. The bars indicate the gain or loss accuracy compared linear probing and VPT, respectively. (a) Compared with linear probing, EVP outperforms linear probing on out of datasets by 2.1% on average. (b) Compared with VPT, EVP beats VPT on 10 out 12 datasets by 5.6% on average.
Table Performance of non-CLIP models. EVP' indicates that train EVP using classes after preproccssing stage, EVP slightly exceeds VP and EVP' outperforms EVP and VP by large margin. The bold indicates the cases that the performance EVP" competitive with linear probing.
Model
Adaptation
CIFAR100
CIFARIO
Flowers
Food
EurSSAT
SUN
SVHN
Pets
DTD
RESISC
CLEVR
Avg.
Instagram
VP
16.7
62.1
4.8
6.5
86.1
2.2
53.8
18.6
29.1
40.6
30,9
31.9
Instagram
EVP
13.6
67.2
9.2
7.1
85.6
7.9
50.8
16.3
29.0
38.1 0
48.1
33.9
Instagram
EVP"
60.3
93.5
11.4
8.4
88.7
19.6
55.3
74.4
44.4
47.5
50.5
50.4
Instagram
LP
61.0
90.1
92.7
65.8
95.5
58.1
48.0
91.5
70.9
95.7
30.2
73.2
Instugram
FT
77.8
77.8
94.5
75.6
97.4
56.7
96.8
93.9
73.5
93-4
87.9
84.1
RNGO
10.1
54.3
4.7
5.1
80.7
1.1
57.1
10.8
8.2
28.3
29.3
26.4
RN50
EVP
9.2
55.9
6.6
3.9
10.5
18.7
35.4
35.5
RNG0
EVP'
21.9
77.0
11.9
7.0
81.0
14.7
72.0
11.2
39.2
37.2
87.7
92.7
57.5
91.1
66.7
32.6
RN50
FT
79.9
94.1
96.9
73.2
96.5
55.9
92.3
66.7
93.4
89.3
84.3
To further evaluate the effectiveness, we next compare EVP with linear probing, which is a widely used fine- tuning protocol. The results, shown in Table 13 and Figure 3 demonstrate that EVP outperforms linear probing on out of 12 datasets. On average, EVP achieved an accuracy 82.5%, which is 2.2% higher than linear probing. In addition, our method is more flexible the number of parameters can be easily controlled (via adjusting the padding size), whereas the number of parameters in linear probing must depend the number of class categories in the downstream tasks.
Lastly, Our method, EVP, exhibits promising performance compared to fully fine-tuning while being sig- nificantly more parameter-efficte performance gap between these two mothods, with an average accuracy of 82.5% for EVP and 88.1% for fully fino-tuning, EVP outperforms achieves similar results on certain datasets, including CIFAR100, CIFAR1O, FuroAAT, DMLab, and Pets.
with only 0.04% the number parameters. While there still a

---

Published in Transactions on Machine Learning Research (05/2024)
Table 3: Robustness comparison out-of-distributtio and corruption datasets. Teft: out-of-distributiin datasets. Right: corruption datasets. We observe that EVP achieves much stronger robustness both out-of-distribution setting and corruption setting.
Model Adaptation
iwildram
ramelyon17
finow
Avg.
Model
Adaptation
CTEAR100-C
CIFARIO-C
Avg.
CLIP VP
57.3
91.4
37.8
62.2
CLIP
VP
52.5
78.3
65.4
CLIP VPT
58.8
91.9
29.7
60.1
CLIP
VPT
54.0
70.2
62.3
CLIP
EVP(Ours)
64.9
95.1
40.2
66.7
CLIP EVP(Ous)
58.6
84.3
71.5
CLIP
LP
66.7
86.0
36.3
CLIP
LP
56.9
78.8
67.9
CLIP
FT
49.7
CLIP
FT
61.1
82.7
71.9
4.2 Non-CLIP Models
In this section, we evaluate the effectiveness of EVP on non-CLIP models. One challenge for adapting non-CLIP models to downstream tasks is that their original classification head is either less semantically meaningful or mapped to a set of predefined classes. direct and naive solution used in VP is to arbitrarily map downstream classes to pre-trained classes and discard all unassigned classes. However, we posit that there could exist some similarity between pre-trained and downstream classes, even if an exact correspondence is not known. This motivates us propose prepprocsssing stage before implementing visual prompting utilize this potential similarity.
Specifically, for each downstream class, we feed downstrram images in that class into the pre-trained model and investigate the predictions in the pre-trained classes. We then simply choose the pre-trained class with the highost prediction frequency as the corresponding class for the downstream class. After pre-processing, we fix the correspondence and train our visual prompting method.
Overall, the results in Table demonstrate the effectiveness of our proposed EVP method for adapting non- CLIP models to downstream tasks. While the performance of EVP is already improved over the bascline VP method (by 2%) when using arbitrary mapping, the use of our pre-processing stage substantially enhances the performance of EVP further, i.e., from 33.9% to 50.4% with the Instagram pretraining model (Mahajan et al. 2018) and from 28.6% to 41.3% with an ImageNet-train ResNet-50. However, we note the performance of EVP" is not as strong on fine-grained datasets, such as Flowers102 and Food101, suggesting that it may be more challenging to find correspondence between pre-trained and downstream classes for these types of tasks.
4.3 Robustness
In this section, we investigale the robustness of EVP compared with other prompting methods and fine- tuning protocols. We evaluate the robustness of EVP out-of-distribation (OOD) and corruption datasets.
OOD robustness. We test the robustness EVP to distribution shift using the WILDS benchmark Koh et al. 2021). The model is trained on datascts from a specific domain and then evaluated on datascts from different domain, such as images from different regions, cameras, and hospitals. The results in Table 14 show that EVP outperforms other prompting methods by at least 4.5%. Additionally, we find that EVP outperforms both linear probing (+3.7%) and fully fine-tuning (+0.7%) in this setting, highlighting the potential of EVP in handling out-of-distribution samples.
Robutness on corruption datasets. In this study, we also evaluate the robustness of EVP to common image corruptions (Hendrycks & Dietterich 2018). We test EVP the CIFAR100-C and CIFAR1O-C corruption datasets, which apply 19 common image corruptions to the CIFAR100 and CIFARIO datasets, respectively. We train EVP on the CIFARI00 and CIFAR10 datasets and then evaluate its performance on the corresponding corruption datasets.
The average accuracy is reported Table 14 (the accuracy under each type of corruption reported in the supplementary material), where we can observe that EVP outperforms other prompting methods and linear probing by large margin. It is also worth noting that EVP performs comparably to fully fine-tuning in

---

Published in Transactions on Machine Learning Research (05/2024)
Average on 11 datasets
EuroSAT
OxfordP ets
SUN
78
89
73
Scores(
68
H1
Scorest (%)
Scoress
77 80 83
Scores(%))
66
64
07
(Our)))))
05
83
74
03
VP
71
robing
56
60
30
1%
4%
7%
10%
1%
4%
10%
Percent training examples per class
Percent training examples per class
Percent of training examples per class
Percent of training examples class
CIFAR100
SVHN
CIFAR10
Resisc
64
SM
80
93
Scores(
76
Scores(
02
Scores
72
68
8
60 72
64
68
63 66
60
88
87
40
4%
7%
10%
4%
10%
Percent training examples per class
Percent training examples per class
Percent trainnng examples per class
of examples per class
Food101
DTD
clevr
DMLab
46 48 50
50
82
Scores
04
Scores
54 57
(cores(%)
44
80
41
76 78
35 38
42 40 51
38 40
16
10%
15
4%
7%
10%
1%
4%
7%
10%
156
7%
Percent of training examoles per class
Percent of examples per
training examples per class
Percent of training examples per class
Figure Results with different data scales on 11 visual recognition datasets. Each figure shows the results trained on 1%, 4%, 7%, and 10% data, respectively. All visual prompting methods show clear dominance compared with linear probing. EVP (red line) outperforms other methods by large margin average.
handling common image corruptions, i.e., 71.5% us. 71.9%. This may be due to the fact that the corruptions on the image can damage the performance of other baselines, while our strategy of treating the prompt as a standalone and independent component can alleviate this issue, i.e., the prompts in EVP are not directly combined with (but are padded around) the corrupted images during inference.
4.4 Different Data Scales
In this section, we evaluate the performance of EVP with different data scales, We train EVP using only 1%, 4%, 7%, and 10% of the data for each class the training datasets. This of particular interest because few-shot learning an important aspect of prompting in natural language processing. We hereby aim validate if EVP can achieve strong performance with limited data.
The average accuracy, well as the detailed accuracy on each dataset, is reported in Figure We can observe that 1) visual prompting methods (VP, VPT, EVP) consistently outperform linear probing, demonstrating the effectiveness of visual prompting in learning with limited labeled data; and 2) among all visual prompting methods, EVP consistently achieves the best overall performance, demonstrating its strong generalization ability at different data scale.
5 Discussion
5.1 Original Image Information
We first investigate the importance of preserving the original information of the input image. To explore this, manipulate the size of the input image while keeping the number of learnable parameters constant.
8

---

Published in Transactions on Machine Learning Research (05/2024)
Table 4: Ablation on original image information across 12 datasets CLIP. There a clear trend that as the image size decreases, L.., as the occlusion decreases, the performance gradually increases.
Image size
CIFAR100
CIFAR10
Flowers
Food
EuroSAT
SUN
DMLab
SVHN
Pets
DTD
RESISC
CLEVR
Avg.
224
77.1
94.7
80.6
$1.8
96.7
71.4
60.1
88.9
89.1
64.2
88.1
73.4
80.5
204
77.5
95.3
80.8
82.0
97.0
71.0
60.7
89.7
89.3
64.0
88.5
73.6
80.8
184
79.0
95.9
81.7
82.3
97.4
70.3
61.9
90.6
89.9
65.2
89.5
74.3
$1.5
164 (Default)
81.2
96.6
82.3
82.3
97.6
71.0
62.3
90.5
88.7
68.4
89.7
75.9
82.2
Table Ablation on the positional embedding at the pixel level. EVP-small shrinks the original image and pads it with learnable pixels to the original size, while EVP-big pads pixel patches around the original image. We note EVP-small w/ PE can beat EVP-big w/o PE despite having fewer parameters and smaller input resolution, suggesting positional embeddings are crucial in visual prompting at the pixel level.
Image size
CIFAR100
CIFAR10
Flowers
Food
EurSAT
SUN
SVHN
Pets
DTD
RESISC
CLEVR
Avg.
EVP-small w/ PE
81.2
96.6
82.3
84.1
97.6
71.0
90.5
90.0
68.4
89.7
75.9
84.3
EVP EVP-smal small w/o PE
70.9
92.3
70.2
78.6
83.0
62.2
55.9
88.1
62.5
73.8
72.1
73.6
EVP--big w/ PE
81.4
96.9
92.3
84.7
97.4
71.8
90.7
89.2
68.9
91.6
77.1
85.6
EVP-big w/o PE
73.4
93.7
71.0
80.7
85.3
63.4
58.1
88.9
64.6
76.2
74.5
75.4
Table 6: Ablation on the positional embedding at the token level. VPT ouly adds positional embeddings to the imaaee patch embeddings, while VP1T, T, VPsu'T denote methods in which the 1st, 25th, and 50th positional embeddings are added to the learnable takens, respectively. We can observe that simply adding positional embeddings to the learnable tokens
can significantly improve performance.
Image size
CIFARI00
CIFAR1O
Flowers
Food
EuroSAT
SUN
SVHN
Pets
DTD
RESISC
CLEVR
Avg.
VPT
76.6
95.0
76.2
84.7
94.6
69.3
86.1
92.1
61.6
84.3
58.6
79.9
VP,T
77.3
96.0
77.5
84.9
96.2
69.7
87.3
92.2
67.7
87.1
59.9
81.4
VP
76.8
95.5
76.3
84.0
95.9
69.0
85.4
92.0
66.6
86,1
58.8
80.6
VP
77.0
96.0
75.4
83.8
95.8
69.3
86.1
92.3
66.4
84.0
59.0
80.4
By scaling the size of the imput image beyond 164, we expect to see an increased overlap between the input image and the prompt. The results are reported in Table 4 A notable observation is that for 9 out of the 12 datasets examined, the performance is inversely proportional to the extend overlap. This underscores the importance of preserving original image information.
5.2 Prompting Positional Embedding
We next probe positional embeddings. Note while EVP encodes the positions of both learnable visual prompts and image patch embeddings, VPT restricts positional embeddings (PE) solely for the image patch embeddings. To illuminate the implications of positional embeddings in visual prompting, we present evaluations across different configurations, both at the pixel and token levels.
Focusing on the pixel level, we denote our main method as EVP-small w/ PE, and introduce two variations: 1) EVP-big w/ PE: Here, the image at the original size is padded with learnable pixels, and the positional embeddings are added to both the image patch embeddings and the learnable pixels using interpolation. 2) EVP-big w/o PE: we hereby also use the image at the original size, but only add positional embeddings the image patch embeddings. Table 5 shows the results. We can obscrve EVP-big w/ PE achieves the best performance, while EVP-big w/o PE performs the worst. For example, we note even EVP-small w/ PE is able to outperform EVP-big w/o PE by an average of 6.1% the five datasets (86.7% us. 78.6%), despite having fewer parameters and lower image resolution. These results demonstrate the vital role of adding positional embeddings in visual prompting.
At the token level, we find that simply adding positional embeddings to the learnable tokens can improve performance. To investigate this further, we create different prompting choices at the token level by adding different positional embeddings to the learnable tokens. Specifically, we denote prompting choices as VP, T, where the n-th positional embeddings added to the learnable tokens, as illustrated in Figure Since the
9

---

Published in Transactions on Machine Learning Research (05/2024)
Visual Prompt
Image Patch
Visual Prompt
Image Patch Embeddings
CLS
Embeddings
CLS
Po
2
33
P4
P3
Po
Pp
P
P
P
2
P
P
Ps
Transformer Encoder Layer
Transformer Encoder Layer
Transformer Encoder Layer
Transformer Encoder Layer
(a) VPT
(b) VP,
Figure 5: Ablation positional embedding the token level. (a). Visual Prompting Tuning(VPT): Inject learnable tokens between CLS and image patch embedding without positional embedding (b); VP, T: Inject learnable tokens between CLS and image patch embedding with positional embedding Pn. (i.e, a-th positional embedding (n 1, ...,, 5)).
Table Ablation on augmentation and normalization. Left panel: We find that simple techniques like RandomF'lip and RandomCropp achieve strong results, while stronger augmentations like Cutmix or RandAug decrease the performance. Right pancl: We note 1) applying L2 norm on gradient enhance performance; and 2) using the whole immage's ' gradient leads to further improvement compared to using only the gradient of the visual prompting pixcls.
Gradient Normalization
Augmentation
Performance
Performance
L1
Loc
L2-partial
Lg-whole
Flip&orop
RandAug
CutMix
77.5
80.5
x
77.2
X
81.2
79.4
X
X
X
V
x
x
71.9
79.4
X
79.7
X
X
81.2
total token count is 50, in our experiment, choose (1, 50] to study the effects of inserting tokens at the head, the middle, and the tail of the original image.
The results in Table 0 demonstrate that the incorporation positional embeddings in the learnable tokens can consistently and significantly improve the performance of visual prompting. For example, the average accuracy is increased by +2.4% with VP,T, +1.8% with VP2sT, and 1.4% with VP T. Collcctively, these findings reiterate the importance of positional embeddings in this context.
5.3 Training strategy
In this section, we turn our attention to the impact of various "old tricks" in transferable adversarial learning on the performance of visual prompting. We first ablate different augmentation methods such as RandomHor- izontalFlip, RandAug, and Cutmix on the CIFAR100 dataset. As shown in Table 7 interestingly, we find that using simple augmentation techniques like GandomHorizontalln satisfactory results, while more advanced methods such as Cutmix or RandAug may docrease performance, likely due to over-regularization. For example, on CIFAR100, RandommorizzoaalPii and RandomCrop is able to improve accuracy by 0.7%, but RandAug CutMix hurts the accuracy by 1.1% and 0.8%, respec- tively.
and RandomCrop can already achieve
Next, we ablate different gradient normalization strategies, including the L1 norm, La norm, and Loo norm, on the CIFAR100 dataset. The results, shown in Table 7 indicate that the L2 norm consistently performs the best among all three strategics. Furthermore, we observe that employing the entirety of the image's gradient to compute the norm (Lz-whole) performs consistently better than merely using the gradient of the visual prompting pixels (Ta-partial), i.e., 81.2% us. 79.4%.
10

---

Published in Transactions on Machine Learning Research (05/2024)
Table 8: Performance of VIT models. By incorporating the output transform into the framework, observe
substantial improvement.
Model Adeptation
CIFAR100
CIFAR1O
Flowers
Food
EuroSAT
SUN
SVHN
Pets
DTD
RESISC
CLEVE
Avg.
VIT VP
44.6
94.9
15.3
36.5
95.3
2.2
16.2
8.8
16.8
64.8
35.7
39.2
VIT
VP+ILM
73.8
97.0
36.9
42.6
96.0
17.6
89.7
79.5
36.2
69.6
64.0
64.4
VIT EVP
73.8
97.7
78.6
62.7
97.6
5.7
19.6
8.7
57.4
89.5
50.2
58.3
VIT
EVP+ILM
86.3
97.6
77.5
70.3
97.6
26.2
95.2
86.6
60.1
89.8
71.2
78.0
5.4 VPT-DEEP
VPT-DEEP is an advanced version of VPT, which additionally introduces learnable tokens at every Trans- former block for enhancing performance. Following the setup in Section 4 we hereby draw a comparative performance analysis between VPT-DEEP and EVP, focusing on the following three settings: the CLIP- based model, out-of-distribution scenarios, and data corruption scenarios. Regarding CLIP-based model, our results show that, while VPT-DEEP outperforms the vanilla. VPT by 4.6% (from 77.3% 81.9%) with significantly more parameters 0.092 million, its performance remains slightly lower than that of our EVP, which achieves average accuracy of 82.5%. In addition, EVP consistently achieves superior performance than VPT-DEEP in more challenging settings, specifically, out-of-distribution (66.7% 65.2%) and cor- ruption (71.5% us. 69.5%). These findings confirm the efficacy of EVP as superior prompting strategy. For more detailed accuracy scores across individual datasets, please refer to the supplementary material.
5.5 Compatibility with label mapping strategy
Though this work only focuses on optimization on input pixels, it is also compatible with other techniques. For examples, label mapping strategy an elfective method in visual prompting, which automatically maps the source labels to target domain labels, in order to mitigate the uncalibrated final layer (i.e., classification head) of non-CLIP models. In order further enhance the performance of visual prompting, this section delves into the in-depth exploration of the compatibility between EVP and ILM. Specifically, we conducted experiments under ViT-Base 16 as the backbone, and results shown in Tab. g We observe that the integration of ILM results in substantial performance improvement of both VP and EVP. In the case of VP and EVP, ILM achieved performance improvement of 25.2% (39.2% to 64.4%) and 19.7% (58.3 % to 78.0 %) respectively. Moreover, irrespective of the use of ILM, EVP consistently outperforms VP.
6 Conclusion
We propose EVP, simple and effective method for adapting pre-trained models to various downstream tasks using visual prompts at the pixel level. EVP preserves the original image information and incorporate adversarial learning techniques improve performance. Our experiments demonstrate that EVP outper- forms other visual prompting methods and outperforms linear probing in variely of settings. Moreover, EVP shows strong performance in handling data of varying scales and robustness against out-of-distribution samples.
References
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring Visual Prompts for Adapting Large-Scale Models. In arriv, 2022.
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. BETT: BERT Pro-Training of Image Transformers. In TCLR, 2021.
Charles Beattie, Joel Z Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq, Simon Green, Victor Valdés, Amir Sadik, et al. Deepmind lab. In arXiv, 2016.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Smndié, Pavel Laskov, Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. Tn ECML-PKDD,, 2013.
11

---

Published in Transactions on Machine Learning Research (05/2024)
Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXic:2108.07255 2021.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food- mood-10-mminin discriminative components with random forests. In ECCV, 2014.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Tn NeurIPS, 2020.
Aochuan Chen, Yuguang Yao, Pin-Yu Chen, Yihua Zhang, and Sijia Liu. Understanding and improving visual prompting: A label-mapping perspective. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 19133-19143, 2023.
Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. In Proc. IEEE, 2017.
Mircea Cimpoi, Subhransu Maji, lasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In CVPR, 2014.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR workshops, pp. 702-703, 2020.
Leyang Cui, Yu Wu, Jian Liu, Sen Yang, and Yue Zhang. Template-Based Named Entity Recognition Using BART. In ACL-IJCNLP, 2021.
Nilesh Dalvi, Pedro Domingos, Sumit Sanghai, and Deepak Verma. Adversarial classification. In KDD, 2004.
Jacob Devlin, Ming-Woe Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, In arziv, 2018.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Xiaolin Hu, and Jun Zhu, Discovering Adversarial Examples with Momentum. In CVPR, 2017.
Yinpeng Dong, Pangzhou Liao, Tianyu Pang. Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. Boosting adversarial attacks with momentum. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018.
Yinpeng Dong, Tianyu Pang, Hang Su, and Jun Zhu. Evading defenses to transferable adversarial examples by translation-invariant attacks. In CVPR, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale. In ICLR, 2020.
Gamaleldin F Elsayed, Ian Goodfellow, and Jascha Sohl-Dickstein. Adversarial Reprogramming of Neural Networks. In ICLRR 2018.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making Pre-trained Language Models Better Few-shot Learners. In ACL, 2021.
lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and Harnessing Adversarial Exam- plcs. In ICLR, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual lcarning for image recognition. In CVPR, 2016.
Kaiming He, Xinlci Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked Autoencoders Are Scalable Vision Learners. Tn CVPR, 2022.
12

---

Published in Transactions on Machine Learning Research (05/2024)
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. In IEEE J-STARS, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. In ICLR, 2018
Ling Huang, Anthony D Joseph, Blaine Nelson, Benjamin IP Rubinstein, and J Doug Tygar. Adversarial machine learning. In Proceedings of the 4th ACM workshop on Security and artificial intelligence, 2011.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual Prompt Tuning. In ECCV, 2022.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Tawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsubramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, et al. Wilds: A benchmark of in-the-wild distribution shifts. In ICML, 2021.
Alex Krizhevsky, GeofTrey Hinton, et al. Learning multiple layers of features from tiny images. In Cileseer, 2009.
Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parameter-Elficient Prompt Tuning, In EMNLP, 2021.
Xiang Lisa Li and Percy Liang. Prefix- Tuning Optimizing Continuous Prompts for Generation. In ACL- IJCNPP, 2021.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. In arriu, 2021a.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. In arXiv, 2021b.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint ar Xrv: 11692, 2019.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards Deep Learning Models Resistant Adversarial Attacks. In ICLR, 2018
Dhruy Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. In NewrIPS, 2011.
Maria-Eloan Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In ICVGIP, 2008.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In CVPR, 2012.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexan- der Miller. Language Models as Knowledge Bases? Tn EMNIP-IJCCNLP, 2019.
13

---

Published in Transactions on Machine Learning Research (05/2024)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Nla Sutskever. Learning Transferable Visual Models From Natural Language Supervision. In PMLR, 2021.
Yun-Yur Yun Tsai, Pin-Yu Chen, and Tsung- Yi Ho. Transfer learning without knowing: Reprogramming black- box machine learning models with scarce data and limited resources. In International Conference Machine Learning, pp. 9614-9624. PMLR, 2020.
Hsi-Ai Tsao, Lei Hsiung, Yu Chen, Sijia Liu, and Tsung-Yi Yi Ho. Autovp: An automated visual prompling framework and benchmark. In ICLR, 2024.
Xiaosen Wang, Xuanran He, Jingdong Wang, and Kun He. Admix: Enhancing the transfcrability of adver- sarial attacks. In ICCV, 2021.
Weibin Wu, Yuxin Su, Michael R Lyu, and Trwin King. Improving the transferability of adversarial samples with adversarial transformations. In CVPR, 2021.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan L Yuille. Improving transferability of adversarial examples with input diversity. In CVPR, 2019.
Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In ICCV, 2019.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. In IJCV, 2022.
A Implementation details
Our implementation is based on Pytorch Paszke et al. 2019). We use CLIP-B/32, Instagram MMhhjann ett al. 2018), and ResNet50 (He et al. 2016) as our pre-trained model, and the batch size 256, 32, 128, respectively. All visual prompting in our experiments are trained for 1000 epochs. For EVP, we use SGD with cosine learning rate schedule; the initial learning rate is 70. The prompting size is 30 pixels by default. To fairly compare with VP, we follow its text prompting setup (Bahnng et al. 2022 in CLIP model. Specifically, we use "This is a photo of a [LABEL]" by default for the text prompting. For CLEVR datasets, we use "This is a photo of [LABEL] objects", for DMLab datasets, we use "The distance is [LABELI]. and the reward is [LABEL2], and for Camelyon17, the text prompting template is "a tissue region L.BEL tumor".
B Prompting size
The prompting size defined as p= K-k where k is the image size after shrinking, and is the input size of pre-trained model. Therefore, the number of parameters is 12p(K p), which only depends on p since K is fixed for given model. In our experiment, the optimal prompting size varies across datasets, shown in Figure G Since we shrink the original image and pad learnable pixels around it, there shows trade-off between the image resolution and the number of parameters. Interestingly, we note that, for datasets with a low resolution (e.g., CIFAR100), the prompling of p = 30 achieves the best performance. While for datasets with a high resolution, we note setting to a small value empirically works the best. For example, we find that p = the best in Food101 dataset which has a resolution of 512 x 512. The best hyper-paramcter is shown in Table 12 and we note setting prompting size to 30 generally achieve the best overall accuracy on these four datasets.
14

---

Published in Transactions on Machine Learning Research (05/2024)
Table 9: The optimal prompting size in experiments across 12 datasets CLIP.
Image
CIFAR100
CIFARIO
Flowers
Food
EurSAT
SUN
DMLab
SVHN
Pets
DTD
RESISC
CLEVR
Prompt size (p)
30
30
30
30
30
30
20
30
30
Top-1 Accuracy on llassificaaion datacets
EuroSAt
58
OxfordP
52
CIFAR100
46
Food101
40
10
20
30
50
70
90
Prompting Size
Figure Ablation on prompting size. The pre-trained model CLIP-B/32. We vary the prompting size, which determines the number parameters, and show the performance on four datasets.
Table 10: Specific performance on CIFAR-10C
Methods
brightness
contrast
Eng
gnussian
gaussia
glass
blur
impulse
VP
91.9
87.8
60.7
61.9
VPT
87.6
78.0
82.0
73.9
78.8
77.1
74.9
46.8
46.0
55.8
O0us)
95.1
90.3
93.1
87.7
91.5
89.8
91.6
66.3
69.0
67.8
LP
92.9
85.0
88.6
82.1
87.0
85.6
86.2
56.2
55.1
02.1
FT
91.9
91.6
92.1
84.2
91.2
88.5
91.1
57.1
59.5
08.7
Methods
joeg compression
mottion bluz
pizelate
satuate
sbot modse
80OW
spaceer
speckle noise
20om blur
Avg.
VP
74.3
79.7
66.8
$8.9
67.5
84.8
87.3
69.1
84.8
78.2
VPT
63.9
58.1
75.9
80.4
86.3
75.3
93.2
73.8
89.6
91.0
75.2
90.3
84.3
L.P
75.1
77.9
85.2
78.8
FT
74.3
86.8
77.3
93.0
66.4
80.0
91.0
69.4
89.0
82.7
Table 11: Specific performance on CIFAR- 100C
Adaptation
brightums
coutrast
defocus blur
elastic
fcoet
gaussian
blur
gaussion
poise
blux
impulse
VP
71.3
55.5
65.7
57.9
61.5
$8.0
30.6
27.8
37.4
VPT
72.8
62.8
67.9
57.9
64.6
58.7
65.4
30.3
25.8
41.0
EVP(O
77.7
66.5
73.2
63.5
69.1
64.2
70.2
36.3
31.8
38.5
I.P
75-5
G3.1
69.8
60.0
66.2
63.4
66.4
31.1
30.8
39.4
FT
31.9
Adaplation jong compression
mationn_
pixelate
siluate
moise
80OW
spakler
speckle noise
z0omL blur
Avg.
VP
17.4
55.2
46.1
61.7
37.7
39.3
63.5
38.
61.1
52.5
VPT
45.2
57.8
45.0
64.3
38.0
60.5
39.4
63.4
54.0
EVPs)
52.8
63.5
51.4
69.1
43.8
65.1
68.6
14.6
68.0
58.6
LP
49.5
61.4
57.2
66.8
39.7
64.7
67.8
42.4
65.8
56.9
FT
18.3
66.5
52.6
73.5
12.9
69.1
72.9
44.3
70.9
61.1
C Comparative Analysis of Model Size, Training Time, and Throughput
In order to provide more comprehensive and in-depth comparison, we conducted experiments on CIFAR100 and listed the training time, throughput, and the number of parameters of vp, vpt, evp, and linear probe, respectively. showed the results.
15

---

Published in Transactions on Machine Learning Research (05/2024)
Table 12: Model Size, Training Time, and Throughput of different methods
Adaptation
Number of parameters (Millions)
Training time (mins)
Throughout (Images/second)
VP VPT
0.070
358
780
0.064
1050
651
EVP
0.062
409
961
Linear probing
0.051
356
1130
Table 13: Performance comparison across 12 datasets with CLIP. We note EVP outperforms VPT-DEEP with fewer tunable parameters. The results where EVP outperforms VPT-DEEP are highlight in bold.
Adaplation Tiaable parans (M)
CIFAR100
CIFARIC
Flowers
Food
EurSAT
SUN
DMLab
SVHN
Pets
DTD
RESISC
CLEVR
Arg.
VPT-DEEP
0.052
78.3
96.1
84.4
85.8
97.4
70.2
57.7
90.1
92.5
70.1
90.6
69.7
8L.9
EVP(Ours]
0,062
96.6
82.3
84.1
71.0
62.3
90.5
80.7
76.9
82.5
Table 14: Robustness comparison on out-of-diitribuution corruption datasets. Left: out-of-distribution datasets. Right: corruption datascts. We can observe that EVP achieves much stronger robustness both ont-of-distribation setting and corruption setting.
Model
Adaptation
iwildeum
camelyon17
fnoww
Avg.
Model
Aduptation
CIFAR100-C
CIFARIO-C
Avg.
VPT-DEEP
62.7
93.6
39.3
CLIP
VPT-DEEP
69.5
EVP(Ours)
61.9
95.1
40.2
CLIP
EVP(Ours)
84.3
71.5
Performance under different corruption cases
In Section 4.3, we see that our EVP outperforms other methods on corruption setting. Here, we list the generalization performance of all methods under various types of corruptions, as shown in Table I0 and Table 11 Specifically, compared to VP, we note 1) on CIFAR-10-C, EVP yields the largest improvement on constrast (+8.8%) and the smallest improvement on brightness(+3355% 2) on SIFAR-100-C, EVP yields the largest. improvement contrast (+11.0%) and the smallest improvement impulse noise (+1.1%).
E Performance comparison with VPT-DEEP
VPT-DEEP is an advanced version of VPT, which additionally introduces learnable tokens at every Trans- former layer's input space for enhancing performance. We hereby briefly compare its performance to that of EVP. Specifically, we compare the performance of EVP and VPT in three settings: model,, OOD, and corruption.
E.1 Performance on CLIP-model
In this section, based on CLIP model, we conduct a comparative analysis of the performance of EVP and VPT-DEEP on 12 classification datasct. The results are shown in Table I We can see that EVP demonstrates an average performance improvement. of 0.6% over VPT-DEEP(32.5% w.s. 81.9%), with only 0.062 Million tunable parameters, which is 0.03 million fewer than VPT-DEEP.
E.2 Performance on Robustness
Following the setting in main text, we test the robustness of EVP and VPT-DEEP to distribution shift and common image corruption. Table 14 presents the specific comparative results. In both the OOD (+1.5%) and corruption settings (+2.0%), EVP achieves superior performance consistently compared to VPT-DEEP, which demonstrates the robustness of EVP.
16

---

Published in Transactions on Machine Learning Research (05/2024)
F More ablation study results
In this section, we demonstrate more ablation study results of augmentation strategy and normalization strategy. As shown in Tab. 15 data augmentation and normalization play a great role in visual prompting,
Table 15: Ablation on augmentation and normalization. Simple techniques like RandomFlip and RandomCrop achieve strong
results, and applying uormalization on gradient enhance performance significantly.
Adaptation
CIFARI00
CIFAR10
Flowers
Food
EuroSAT
SUN
SVIN
Pets
DTD
RESISC
CLEVR
Avg.
EVP w/o augmentation
80.5
96.2
83.5
68.0
79.4
67.8
74.6
$2.3
EVP w/o normalization
77.5
95.7
78.5
80.9
92.9
51.3
72.3
78.7
18.2
82.6
72.1
75.8
EVP
81.2
96.6
82.3
84.1
97.6
71.0
90.5
90.0
68.4
89.7
75.9
84.3
17


Reviews:
[{"review_id": "dEcoPhlvhQ", "reviewer": "Reviewer_QkpB", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "###  Strengths:\n* The idea is easy to follow.\n* The perspective on the connection between adversarial examples and visual prompting is interesting.\n* Ablation results are interesting, for example the importance of positional embedding.\n\n### Weaknesses:\n* The source of the improvement is not clear. For example, I would wanna know if the diversified input and the gradient normalization can be applied to VP and obtain some improvement.\n* Although there are some interesting observations, the discussions are not thorough. For example, would it be much better to introduce some prompts in the middle of the image, if the positional embedding is important?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors proposed a new visual prompting method called Enhanced Visual Prompting (EVP), that is motivated from the Visual Prompting and combined with different training strategy like diversified input and gradient normalization. The authors provide thorough experiments to demonstrate the effectiveness of the proposed method.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like to see a more systematic study on the relevant training strategies, as I mentioned in the weakness part.", "broader_impact_concerns": "N/A.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bCYPwyZWmS", "reviewer": "Reviewer_WuQV", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n\n1. The paper is easy to follow and well-written.\n2. The experiments are complete and solid.\n\nWeaknesses:\n\n1. Some motivations and explanations of the proposed method are needed. For example, why do you use gradient normalization? Are there any findings related to the tricks used, like augmentation and gradient normalization, that are special to visual prompts?\n2. The novelty is a limitation since the tricks are all old ones. However, novelty is not a focus of this venue.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents an improved visual prompt method. The main idea is to first resize the image, then augment the image, and finally train the prompt with gradient normalization. This method is implemented on many tasks and datasets and shows advantages over the classical visual prompt method and linear probing to some degree.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "1, Can you provide a theoretical or intuitive understanding of how the used tricks help the visual prompt method? Are there any new insights into these tricks for the visual prompt method?\n\n2. Can you show the comparison with the Linear Probing method and the Visual Prompt method, in terms of the memory cost, time cost, and throughput?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UEegfw1BQk", "reviewer": "Reviewer_dzmD", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n1. The paper presents a significant improvement over the state-of-the-art visual prompting, with empirical evidence showing an average accuracy increase of 5.2% over previous methods.\n2. Using adversarial training techniques to improve the optimization and generalization of visual prompts is a creative adaptation from another domain.\n\n## Weaknesses\n1. The paper's methodology concatenates three discrete aspects: VPT, augmentation, and normalization. However, it lacks an organic exposition of the relationships between these three points, merely presenting a combination of tricks without elucidating the necessity of integrating the aforementioned three tricks.\n2. The lack of comparison with large-scale datasets such as ImageNet could be a limitation.\n3. The paper could benefit from a deeper theoretical analysis explaining why the proposed method works better than existing approaches, providing insights beyond empirical results.\n4. The ablation studies are helpful, but additional insights into the failure cases or limitations of the method would provide a more balanced view of the EVP's performance.\n5. The method's performance on fine-grained datasets like Flowers102 and Food101 is not as strong, suggesting a potential area for further improvement.\n\n---\n\nOverall, the paper makes a certain contribution to the field of visual prompting and the adaptation of pre-trained models, with clear empirical evidence supporting the effectiveness of the proposed methods. However, the paper is merely a stack of common tricks lacking originality. Additionally, it could include more analysis and benchmarking to strengthen the findings and provide a more comprehensive assessment.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a visual prompting method, Enhanced Visual Prompting (EVP), for adapting pre-trained models to downstream tasks. The authors propose two main innovations: a non-overlapping strategy for combining prompts with images and applying adversarial example techniques (input diversity and gradient normalization) to visual prompting. The method achieves state-of-the-art performance across multiple datasets, outperforming existing methods like linear probing and VPT in terms of accuracy, robustness to distribution shifts, and corruption, as well as flexibility in handling different data scales.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "See weaknesses", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "VtYL43OvdD", "reviewer": "Reviewer_ExA8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. There are comprehensive experiments and comparisons on different configurations (image size, data size, gradient normalization).\n\n2. The discussion on position embedding (PE) is enlightening, and the results show a substantial accuracy gap compared to the absence of PE.\n\n\nWeaknesses:\n1. The proposed method is straightforward. More explanation on why gradient normalization can benefit visual prompting should be provided.\n\n2. In Table 5, only five datasets were selected in the experiments, but the reasons for not selecting the others were not provided.\n\n3. It is not clear how the output transformation is being implemented. Recent works on Visual Prompting have also shown that prompt/image size [1, 2]  and output mapping [1, 3] could affect VP performance. A comparison with them will strengthen the proposed method.\n\n4. As mentioned in Sec. 5.2, the results showed that simple augmentation techniques like RandomHorizontalFlip and RandomCrop can already achieve the best results. It is not clear why the proposed method considers other augmentation in the EVP framework.\n\n5. The performance in non-CLIP models is much worse than LP (Table 2). The author could explore not only CNN-based models (i.e., ResNet-50 and Instagram) but also ViT. \n\n6. The difference in trainable parameter size between EVP-small and EVP-big is not clearly explained. The statement \"EVP-big pads pixel patches around the original image\" in the caption of Table 5 is confusing; it is not clear whether the term \"original image\" refers to the dataset's resolution or the input size required by the model. Also, please also add the results of “EVP-small w/o PE.”\n\n[1] AutoVP: An Automated Visual Prompting Framework and Benchmark (ICLR 2024)\n\n[2] Exploring Visual Prompts for Adapting Large-Scale Models\n\n[3] Understanding and improving visual prompting: A label-mapping perspective (CVPR 2023)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposed EVP (Enhanced Visual Prompting): a visual prompting method that integrates non-overlapping images and learnable prompts. Strategies such as data augmentation and gradient normalization are employed to enhance performance. EVP's reported accuracy, robustness, and generalization ability on limited data surpass the baselines (VPT, VP, and LP). Additionally, the author identifies the significant role of position embedding in training visual prompts, making a comparison with VPT.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to the weaknesses. In addition, please also address the following concerns:\n\n1. The improvement in performance resulting from different techniques (augmentation and normalization in Sec. 5.3) has only been validated on CIFAR100. It is unclear whether these techniques would still effectively enhance accuracy on other datasets.\n\n2. The discussion on different designs of positional embedding could be enhanced. In addition to interpolation and fixed n-th PE, exploring more optimal approaches could provide further insights.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The recommendation is based on the reviewers' comments, the action editor's evaluation, and the authors’ response. 

This paper proposed methods to improve visual prompting (VP) based on enhancing input diversity and gradient normalization, inspired by prior studies on adversarial examples. The results show improved empirical performance (on average and by large) over standard VP. All reviewers agreed that the paper provides solid results and empirical evidence while sharing a common comment on lacking sufficient technical novelty (given that these techniques are known methods in alternative topics). However, the contributions are modest and meet the TMLR Acceptance Criterion (https://jmlr.org/tmlr/acceptance-criteria.html).

Some related and concurrent works on VP were not discussed in the paper.
- Visual prompting was firstly studied in the context of model reprogramming https://arxiv.org/abs/2007.08714
- A recent (or concurrent) work also studied methods to improve VP, especially for the effect of the prompt size. https://arxiv.org/abs/2310.08381


Overall, I recommend acceptance of this submission with minor revision. I expect the authors to expand the discussion on relevant works and include the new results and suggested changes during the rebuttal phase to the final version.
Decision: Accept (poster)