AutoVP: An Automated Visual Prompting Framework and Benchmark
Authors: Hsi-Ai Tsao
Lei Hsiung
Pin-Yu Chen
Sijia Liu
Tsung-Yi Ho
Venue: ICLR 2024 poster (2024)
Keywords: visual prompting
reprogramming
parameter-efficient fine-tuning
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: AutoVP is a comprehensive framework for automating design choices in Visual Prompting (VP) methods. It outperforms existing techniques and serves as both a hyperparameter tuning tool and a benchmark for VP.

Abstract: Visual prompting (VP) is an emerging parameter-efficient fine-tuning approach to adapting pre-trained vision models to solve various downstream image-classification tasks. However, there has hitherto been little systematic study of the design space of VP and no clear benchmark for evaluating its performance. To bridge this gap, we propose AutoVP, an end-to-end expandable framework for automating VP design choices, along with 12 downstream image-classification tasks that can serve as a holistic VP-performance benchmark. Our design space covers 1) the joint optimization of the prompts; 2) the selection of pre-trained models, including image classifiers and text-image encoders; and 3) model output mapping strategies, including nonparametric and trainable label mapping. Our extensive experimental results show that AutoVP outperforms the best-known current VP methods by a substantial margin, having up to 6.7% improvement in accuracy; and attains a maximum performance increase of 27.5% compared to linear-probing (LP) baseline. AutoVP thus makes a two-fold contribution: serving both as an efficient tool for hyperparameter tuning on VP design choices, and as a comprehensive benchmark that can reasonably be expected to accelerate VP’s development. The source code is available at https://github.com/IBM/AutoVP.

Published as a conference paper at ICLR 2024
AUTOVP: AN AUTOMATED VISUAL PROMPTING FRAMEWORK AND BENCHMARK
Hsi-Ai Tsao", Lei Hsiung"' Pin-Yo Yu Chen',Sijia Liu', Tsung- Yi Ho I National Tsing Hua University. 2 Dartmouth College, IBM Research Michigan State University, The Chinese University of Hong Kong Equal contribution
ABSTRACT
Visual prompting (VP) is an emerging parameter-efficient fine-tuning approach adapting pre-trained vision models to solve various downstream image- classification tasks. However, there has hitherto been little systematic study of the design space of VP and no clear benchmark for evaluating its performance. To bridge this gap, propose AutoVP, an end-to-end expandable framework for au- tomating VP design choices, along with 12 downstream that can serve as holistic VP VP-performance benchmark. Our design space covers 1) the joint optimization of the prompts; 2) the selection of pre-trained models, including image classifiers and text-image encoders; and 3) model output mapping strategies, including nonparametric and trainable label mapping. Our extensive experimental results show that Auto outperforms the best-known current VP methods by a substantial margin, having to 6.7% improvement in accuracy; and attains a maximum performance increase of 27.5% compared linear-probing (LP) baseline. Vu thus makes two-fold contribution: serving both as an efficient tool for hyperparameter tuning on VP design choices, and as a comprehensive benchmark that can reasonably be expected to accelerate VP's development. The source code is available at Whttps: / /gghhub.. com/ IBM/ OV?
classifissii tasks
INTRODUCTION
Originating in the domain of natural language processing, prompting (Gao et al., 2021 Lester
et al. [202T] (Shi et al. 2023) has gained considerable popularity as parameter-effciient fine-tuning approach for adapting pre-trained models to downstream tasks. Prompting's methodology has recently been extended to the field of computer vision, where it is termed visual prompting (VP) (Bahng et al.)
2022). In its simplest form, VP can perceived as an in-domain model-reprogramming technique (Chen 2022). More specifically. it adjusts the inputs and outputs of a pre- pre-trained vision model to address downstream maaageclassificatton tasks, without having to make any changes to the weights or architecture of the source model's pre-trained backbone. As such, it stands in contrast conventional transfer-learning methods that involve complete fine-tuning, LP (i.e., involving modifications of the trainable linear layer in the penultimate layer's output), or zero-shot learning (Radford et al.) (2021). For instance, as illustrated in Figure VP adds a universal trainable data frame to the target samples the model-input stage, and then employs a mapping function which can be either explicitly defined or implicitly learned to associate the source and target labels at the output stage.
While VP exhibits tremendous potential, there are two critical challenges that limit its research and development. The first is absence of systematic VP framework. That is, VP design choices, such as prompts' sizes and shapes, source models, and label-mapping (L.M) strategies, have thus far only been studied separately, generally for the purpose of delineating their distinct roles in enhancing downstream task accuracy. Ideally, such a systematic framework would automatically search for the best configurations for performance optimization. For example, [Bahng et al. (2022) have demonstrated that changing the padding size of visual prompts can yield around 15% variation in final accuracy. It has also been observed that VP is better at augmenting large text-image models, such as CLIP (Radford et al. 202I). than pure vision models like ResNet50 (He et al., 2016). In study by Chen et al. (2023b), iterative label mapping (ILM) during training achieved accuracy up

---

Published as a conference paper at ICLR 2024
Trainable Modules
Modules
Scalling
Vioual Promupt
Pre trained Classifier
Fabel Napping
Berkklg
TallyMop
+0.5
detinod
55.2
87.8
83.4
81.5
AutoVP
I.M-VP
Accuracy (%)
67.4
63.7
:2.00
57.5
55,9
51.8
35.3
27.9
24.8
23.9
30.2
14.8
EuroSAT
CIFAR1O
1lo
SVHN
Pets
GTSRB
ISIC
CIFAR100
UCF101
DTD
Foodtur
BMoW
Figure 1: Overview and key highlights of AutoVP. The main components of Au are: Input Scaling, which offers three initial input scale options: x0.5, x x1.0, and x1.5; Visual Prompt, which pads the prompts to the scaled input image; Pre-trained Classifier, allowing users (or Auto to select from four pre- trained models: ResNet18 (He al. 2016). ResNeXt1O1-IG (Mahajan al. 2018). Swin-T (Liu et al. 2021), and CLIP (Radford et al. 2021); and Output Label Mapping. offering four label mapping options: Iterative Mapping (IterMap), Frequency Mapping (areqMap). Semantic Mapping (SemanticMap), and Fully Connected Layer Mapping (FullyMap). Bottom panel: Given a fixed ImageNet-pre-traing classifier (ResNet18) Auto VP outperforms state-of-the-an (ILM-VP in Chen et al. (202b)) on all 12 different downstream image-llassffiation tasks.
13.7% better than fixed label mapping strategies. The second critical challenge is lack of unified performance benchmark: the existing literature evaluates the performance of proposed VP methods in an ad hoc manner, by reporting arbitrarily selected downstream datasets, making comparisons across different methods difficult at best.
To bridge this gap, this paper proposes AutoVP, solution addressing both these challenges via 1) its automated, extendable framework for joint optimization of a) input-image scaling (i.e., prompt size), b) visual prompts, c) source model selection, and output label-mapping strategies; and 2) its provision of a unified benchmark consisting of 12 diverse mage-cllasiffiation tasks with quantifiable content-similarrty relative the dataset (c.g., ImageNet) used for source model pre-training.
As shown in Figure D the first component (i.e., input scaling) of AutoVP determines the optimal ratio between the sizes of prompts and the original images. The second, visual prompts, serve as trainable parameters, and undergo iterative updates during the backpropagation phase. The pre-trained model extracts pertinent features and renders predictions within the source domain; and finally, output label mapping establishes connection between the label spaces of the source and target domains, facilitating accurate predictions in the downstream domain. The modularity of VP allows for the seamless integration and easy extension of various designs for these four components.
Table I compares Au against prior VP proposals and the other two baselines proposed to date: LP and text-prompt (TP)-based zero-shot inference. As the table shows, Auto VP is the only such framework that considers the broad range of settings that can affect VP performance. Moreover, thanks to such settings' collective optimization, Auto VP's configuration amounts to breakthrough in average accuracy across 12 distinct downstream tasks. For instance, with CLIPP as the pre-trained model (see Table 2), AP's average accuracy is 4.6% higher than CLIP-VP's (Bahng al. 2022) and 2.1 % higher than ILM-VP's (Chen et al, (023b). Auto also surpasses LP's accuracy by 0.7% on average, suggesting that it is competitive alternative LP in terms of transfer learning.
We summarize the main contributions as follows:
AutoVP is the first end-to-end VP framework that simultaneously takes account of the design of input scale, visual prompts, pre-trained model selection, and output LM strategies. This modular approach to automating VP gives its users flexibility for VP engineering, as well as a straightforward, comprehensive performance benchmark based 12 downstream image-classification datasets.
2

---

Published as a conference paper at ICLR 2024
Table 1: Comparison of Auto VP with other baselines, including Linear Probing, CLIP zero-shot inference with text prompts (i.e. CLIP-TP in Radford et al. (202T). CLIP-VP (Bahng et al. 2022).
and ILM-VP (Chen al. 2023b). The average accuracy evaluated over 12 downstream tasks (see
Section 4 For detailed information about the setting configurations, please refer to Section
Pre-trained
Output
Output Mapping
Average
Method
Prompt Size
Classifier
Transfornation
Number
Accuracy (%)
Modified Last
Linear Probing
CLIP
79.86
Classification Layer
CLIP-TP
CLIP
Fixed Text Prompt
1
49.54
CLIP-VP
CLIP
30
Fixed Text Prompt
1
76.01
R8
48
45.19
ILM-VP
IterMap
CLIP
30
78.45
ResNet18
IterMap
AutoVP
ResNeXt101-IG
FullyMap
Trainahle
1/5/10
81.02
(Ours)
Swin-T
FregMap
CLIP
SemanticMap
The proposed hyper-parameter tuning process is capable of identifying optimal configurations tailored to individual downstream datasets. In addition, its novel components e.g., automated input scaling (Section B) and weight initialization (Section 4.2 augment VP's overall efficacy significantly, as compared to state-of-the-ar VP methods, LP, and zero-shot baselines (see Table I
This paper represents the first step in a comprehensive exploration of optimal configurations across varied conditions (e.g., fixing source model or output-mapping strategy), and presents an analysis of domain similarity's 's impact on VP performance for each downstream dataset.
This paper highlights AutoVP's superior performance over LP in data-limited settings (Figure and its better nut-of-distribun robustness than LP (Figure
2 BACKGROUND AND RELATED WORK
Background of Visual Prompts. Traditionally, to derive task-specific machine- -eaaning models, researchers have to train update all model parameters. But, amid the advancement of powerful foun- dation models, model fine-tuning and training from scratch have both become time-consuming and parameter-ineffiiient approaches, usually requiring large amounts of training data and storage space. To this end, VP, also known as in-domain model reprogramming. has emerged as an effective means of obtaining learnng models for various Nomainn-specifi tasks (Chen 2022). A well-developed pre-trained model from a source domain can be directly used for performing tasks in the target domain with little transformation of the target data. For example, we can use an ImageNet pre-trained model to classify medical images without modifying any of its parameters (T'sai et al., 2020). On the other hand, VP, along with temperature scaling, can also be used as post-processing calibration method to align model confidence and accuracy (Tang al., 2022 Hsiung al. (203). As compared to traditional approaches such as transfer learning, model fine-tuning, and training from scratch, VP low complexity and model-agnostic strategy; and it is especially suitable for low-data domains.
The Design of Visual Prompts. A VP framework can generally be divided into two trainable modules, one for impul transformation and the other for outpul transformation. These are respectively placed at the input and output ends of a pre-trained model. In the case of input transformation, previous literature has proposed various ways generate and place visual prompts. One of the most popular such approaches is to pad frame around the target task image and then fill it with trainable additive-input perturbation (prompts) Tsai al.) 2020, Chen et al. 2023b Elsayed et al. 2019, Bahng et al.) 2022 Wu et al. 2022 Oh et al. 2023). Next, since the output logits of the source pre-trained model are still in the source domain, further output transformation (e.g.. LM) is required to obtain the target-domain logits. One naive way of achieving this is randomly mapping (RandMap) LM (FreqMap),
m source labels onto the target labels. Tsai et al. (2020) found that frequency- which constructs its LM from the source-laber label prediction distribution of the target-domann data, can further improve the accuracy of downstream tasks. However, Chen et al. (2023b) argued that FreqMap lacks interpretability and that its interaction with VP is difficult to measure. To address that
3

---

Published as a conference paper at ICLR 2024
problem, the authors proposed iterative LM (IterMap), transformation of FreqMap that enables it to concurrently design LM and visual prompts. Yang et al. (2023), meanwhile, proposed a semantics- based LM approach that aligns source and target classes that have similar semantic meanings. And Liao et al. (2023) utilized a prototypical verbalizer to map a mask token to downstream labels, thus providing different perspective on L.M. In this paper, follow similar design to [Bahng al.) (2022). in which visual prompts are placed around images for input transformations, and there are four mapping methods for output transformations. Further details will be presented in Section
Non-universal Visual Prompts. Instead of utilizing universal input prompts, some researchers have focused on designing input-aware prompting models (Zhou et al. 2022a b). For instance, Chen et al. (2023a) generated class-wise visual prompts to improve model robustness. Similarly, SSun
to address accuracy drops caused by low-voltage-indueed bit errors, [unn et al. (2023) proposed an
input-aware add-on module to generate robust prompt; and Loedeman et al. (2022) proposed the
Prompt Generation Network (PGN), which generates visual prompt token vectors based on input
images, allowing for more adaptive and context-aware prompting.
Although input prompting is commonly applied directly to the target image, researchers have also developed other prompting methods, such as convolutional visual prompt (T'sai et al. 2023), which learns prompting parameters in a small convolutional structure through sulf--upervision tasks without knowledge of ground truths, and visual prompt tuning (Jia et al., 2022 Sohn et al.) 2023), which learns prompting parameters at intermediate layers of a source model. In this paper, we focus on a pixel-level VP approach using a task-specific prompting model for each image-llasifficttion task. As such, our approach closely resembles real-world scenarios in which pre- pre-trained source model remains unmodified, and external variations are not introduced internally.
AUTOVP FRAMEWORK
Following the system overview of Auto in Figure T we present its four main components (Input Scaling, Visual Prompt, Pre-trained Classifier, and Output Label Mapping) and its ppprm-pr tuning feature, which enables the joint optimization of these components. Our framework can be extended to support user-defined configurations.
Input Scaling. In our implementation of AutoVP, we choose frame-shape prompts the default prompting template. Hence, the visual prompt sizes p represent the width of the frame, and its actual number of parameters is 2cp(h + w - 2p), where c, w, and h are color channels, width and height respectively. Although the input image size is determined by the source model, when fine-tuning to a downstream dataset from a source model, there is design freedom to resize the target images and use the remaining space for visual prompts. For instance, if the source model takes images with size 224 224 as input, one can scale the target image size to 128 x 128, resulting in the final visual prompt of size p = (224- - 128)/2 (2022) that the prompt size (p) plays a key role in VP performance. To automate the process of optimizing image resizing scale, we design gradient-based optimization algorithm to implement the input scaling module, which is implemented using kornia. geometry. t transform () from the Kornia library (Riba et al. 2020). The transform function integrates range of geometric transformations for 2D images into deep learning, including differentiable image scaling. In addition to image resizing, the prompt size p will also scale up or down fill the remaining space. Furthermore, to facilitate the optimization of image resizing and avoid bad local minima, we set the default image size to 128 along with three initial scales: 0.5, 1.0, and 1.5 to optimize, and the corresponding prompt sizes are 80, 48, and 16 respectively. Consequently, the input scaling module allows A to obtain the optimal image resizing scale and prompt size (p).
48. It was shown in Baann et al. (2022) and Wu et al.
Visual Prompt. For the visual prompt module, AutoVP adds universal pixel-level prompts around all (resized) input images. Let x, & denote the targer (flattened) input image (of N--dimension) ár E N. be the prompted image, which fits the input dimension (N,) of the pre-trained source model fa, (8, denotes its weights), d N.N. be a trainable universal perturbation, and Mp {0, 1) N, be binary mask of prompt size p, indicating the prompting area. Hence, the prompted image 3. can be formulated as:
ã = P(x)) = InputScaling, (z.) + Mp 1 (0))
(I)
Prompts
4

---

Published as a conference paper at ICLR 2024
The prompts are initialized 0 and formally defined as Mp @o( where is the Sigmoid function that maps the input to a value between and 1 (the scaled input pixel value range), ensuring it has the same numerical range as the input image. We then update ó using gradient descent.
Pre-trained Classifier. After applying the prompts to the resized image through the preceding
stages, the prompted image is subsequently fed into the pre-trained model, which serves as a feature extractor to generate predictions in the source domain. We include four representative pre-trained models in our Auto VP framework: ResNet18 (He et al. 2016), ResNeXt1O1-IG (Mahajan 2018), Swin-T (Liu et al. (02T), and vision- Iasion-language multi-modal model, CLIP (Radford et al. 2021) with the VIT-B/32 vision encoder backbone. Note that in Auto the weights of the pre-trained classifiers are frozen and kept unchanged. The details of the models are provided in Appendix A.1
al.
Output Label Mapping. The pre-trained models predict target data to source labels, while the last mile for VP to map predictions the source labels to target classes. As illustrated in Figure [ Vu provides four types of output mapping, and they can be generally categorized into two groups. (i) nonparametric label mapping: frequency mapping (FreqMap) and semantic mapping (SemanticMap), which are defined during the initialization of VP training and remain unchanged throughout the training process; and (ii) trainable label mapping: iterative label mapping (IterMap) and fully connected layer mapping (FullyMap). These two methods dynamically adjust the mapping based on the prompted images. In the following, we provide the overview of our four output mapping approaches, please refer to Appendix A.2 for more details.
Frequency Mapping (FreqMap) is proposed by Tsai et al. (020). It utilizes the source-label prediction distribution of the target-domain data map each target class to the top-m most frequent source classes. Let yd- = {0, ... K, }} and Di = {0, ... K, - } be the set of source and target labels, where K/KK, are the numbers of source/target classes. Consider &G collects all prompted images of label 3t in target domain D,, i.e. X; - - P(x,, (sn, -b)), (Ki,, Me,) E Di), then when m 1, the mapping of y can be defined as:
2/ V/2 = arg max
Σ
Pred (fo, (ār) V6)).
11,
y/ = arg max So, (T:)
Pred(fa. fo. (Z.).
(o.
otherwise
The objective of FreqMap is to map the target label y the source label v,, which is the most frequent label that fa, classified X, as. If source class is selected as the most frequently predicted class for multiple target classes, it will be assigned to the target class that has the highest count of predictions. The general many-to-one frequency mapping algorithm is provided in Algorithm [ in the Appendix [..2 Moreover, random label mapping (RandMap) can be viewed as special case of FreqMap by randomly assigning a subset of source labels to a target label.
Iterative Mapping (IterMap, or ILM) is proposed by Chen al. (2023)). which is iterative approach for updating Frpq IterMap performs the frequency mapping at the beginning of each training epoch to obtain new mapping distribution that aligns with the updated prompts.
Semantic Mapping (SemanticMap) follows the works from Yang al. (2023) and Yen et al. (202I). We utilize the text encoder CLIP to generate the embeddings of the names of the source and target classes. Subsequently, we map the source-target pairs based on the highest cosine similarity score between their respective embeddings. Hence, SemanticMap can be utilized in any of the three vision pre- trained models (ResNet18, Swin-T, and ResNeXt1011IGI by establishing mappings between the target classes and semantically similar classes from ImageNet-IK: However, SemanticMap is not applicable for CL.IP, it does not have explicit set of source domain classes.
Fully Connected Layer Mapping (FullyMap) uses a linear layer to map the source output logits to target classes (Arif et al. 2023). FullyMap can be represented as L, = w Lg + b, where L, is the output logits from the source pre-traine pre-trained model, w and b are the weight and bias vector of the linear layer, and L, is the output of the linear layer which also serves as the final output logits of the VP model.
End-to-end Hyper- Akyper-parane parameter Tuning. Auto overall tuning procedure is depicted in Appendix Given its flexibility and modularity, its users must consider numerous settings (n 222),
A.3
5

---

Published as a conference paper at ICLR 2024
Table Comparison of VP testing accuracy (%) using CLIP as a pre-trained model on 12 datasets; the optimal tuning settings of VP and the final prompts sizes p are also provided. In the Auto VP setting field, the notation "Mapping-m"" represents mapping m source classes to each target class.
Dataset
AutoVP Setting
AutoVP
ILM-VP
CLIP-VP
I.P
SVHN Netzer al.
2011
FullyMap, p 51
92.9 :: 0.2
91.2
88.4
65.4
CIFARIO
Krizhevsky Hinton
2009
IterMap-1, 23
95.2 0.0
94.4
94.2
95.0
Flowers102
Nilsback & Zisserman
2008
FullyMap. p = 16
90.4 = 0.6
83.7
70.3
96.9
Food101 Bossard et al.
2014
FreqMap-1. p 16
82.3 1:: 0.1
79.1
78.9
84.6
UCF101
Soomro et al.
2012
FullyMap, 16
73.1 :: 0.6
70.6
66.1
83,3
Oxf (Parkhi Paakhi al
2012
FreqMap- 10, p 16
88.2 + 0.2
85.0
85.0
89.2
CIFARI00 Krizhevsky & Hinton
2009
FullyMap, 31
77.9 :: 0.6
73.9
75.3
80.0
EuroSAT (Helber et
2019
FullyMap. p = 16
96.8 0.2
96.9
96.4
95.3
DTD (impoi ct al
2014]
FullyMap. p = 17
62.5 = 0.3
63.9
57.1
74.6
ISIC [Codella et al.
2019 Tschandl al.
2018
IterMap-1. 16
74.0 + 1,0
73,3
75.1
71.9
FMoW Christie el al.
2018
FullyMap, 16
40.8 :: 0.8
36.8
32.9
36.3
GTSRB Houhen et al.
2013
FullyMap. 80 93.1 + 0.2
92.6
92.4
85.8
Average Accuracy
80.6
78.5
76.0
79.9
including how big the initial input image should be, whether to trainable resizing module. which pre-trained classifiers to adopt, what output-mapping method to implement, and the number of source labels to map for each target label. To speed up the tuning operation and save computational resources, we use Ray Tune (Liaw et al. (2018) along with an early-stop strategy for terminating poor trails. In our experiments, we employed grid searches test all configurations. An ASHA scheduler (Li et al. 2018) was used to retain the top-n tasks, and we continued training them while stopping the remaining tasks early. We established experimentally that n top tasks were enough to obtain the optimal setting. When the few-epoch tuning process (training 2-5 epochs with each setting) is complete, we select the setting having the highest testing accuracy and conduct complete training using that setting. By using hyper-parameter tuning, VP can efficiently find the best configuration of VP and lead to significant accuracy improvement in downstream tasks.
EXPERIMENTS
Experimental Setup. We evaluated the performance of Au on 12 downstream datasets (CI- FARIO, CIFAR ISIC, SVHN, GTSRB, Flowers102, DTD, Food101, EuroSAT, OxfordIlITPet, UCF101, and FMoW), which are common datasets when measuring transfer learning generalization. Detailed descriptions of these datasets are given in Appendix B.] We repeated each Au experi- ment in triplicate, utilizing a learning rate of 40 with the SGD optimizer for CLIP, and a learning rate of 0.001 with the Adam optimizer for the other pre-trained models. The results of the baselines (CLIP-VP VP (Bahng et al., (022) and ILM-VP (Chen al. (0236)) were extracted from the reported accuracies in their respective papers (please refer to Appendix B.2 for more details). Our experiments were performed on NVIDIA GeForce RTX 3090 and are implemented with PyTorch,
4.1 EXPERIMENTAL RESULTS
Comparison of Vu and Prior Work. To ensure that our comparison of Auto against previ- ously proposed VP approaches was fair, we fixed its source model but relaxed its other hyperparameter tunings. The results of using CLIP as the source model are presented in Table 2 along with the optimal settings arrived at. We compared AutoVP against I.P and two state-of-the-are VP baselines, CLIP-VP VP and ILM-VP, whose configurations can also be found in Table [ With the optimal configu- ration chosen via the tuning process, Auto outperformed these other approaches by up to 6.7% on nine of the 12 target datasets. Additionally, Auto surpassed the LP baseline half those datasets, by a maximum of 27.5% in the case of S SVHN. Auto VP also obtained the best average accuracy.
We observed that Auto VP employed FullyMap as the output transformation most datasets. We speculate that this might have been because the linear layer has more parameters and thus allows the achievement of better results. Also, when Au selected initial image scales, it had a tendency to

---

Published as a conference paper at ICLR 2024
Data Scalablitt
Sepper-praameter Tuning Result (on Flowers102)
Autet Tap-
Pui Tain
80.9
Pull Thenkng Wi
Ave Gat (Lasop
223
Pellylip or
JuaySAy
74.0
79.9
Semunaiclp
70.8
75.6
13.1
68.0
Accuncy
2.7
0.8
1.0
ResNext101
Percentage of Data Usage
The Pre-trained Model of AutoVP
Figure 2: Data Scalability. The chart presents the average accuracy of Au and LP across the 12 datasets with varying data percentages: 100%, 50%, 25%, 10%, and 1 1.. The green bar represents the accuracy gains achieved by Au- toVP compared to Linear Probing (L.P).
The Tuning Result Flowers102 Dataset. The color bars represent the accuracy tuning for 3 to 5 epochs (with ASHA early-stop optimization). Points illustrate the test accu- racy achieved for the best settings given the pre- trained model and mapping method.
Figure 3:
scale up those images with relatively small prompt sizes. This allowed the VP model allocate more attention to the image itself, leading to improved overall performance. As shown in Figure when ResNet18 was used as the source model, Au outperformed ILM-VP by 24.8% average. More experimental results under this setting are provided in Appendix C.I
P with Source Model Selection. We also allowed Au search the optimal source model for downstream tasks. The optimal settings selected by AutoVP, and a comparison of experi- mental results can found in Appendix C.T] Our experimental results show that Swin-T T was the pre-trained model most frequently chosen by Au as most suitable, i.e., in the cases of eight of the 12 datasets. On average, this choice resulted in 0.43% better accuracy than when CLIP was utilized as the fixed pre-trained backbone. On the DTD and Flowers 102 datasets, however, Swin-T's performance was better than CLIP's by much more: i.c., 6.80% and 3.08%, respectively. These findings highlight how multiple pre-trained models can be leveraged enhance performance across diverse range of datasets.
Data Scalability. To understand how AutoVP would perform in data-limited regime, we gradually and uniformly reduced the amount of training data to 50%, then 25%, then 10%, and finally 1% of each training dataset's original size. The experimental results in Figure]innicate that Au consistently outperformed I.P across all 12 datasets, and that its relative performance was especially high in the two scenarios with the lowest data volumes, i.e., 10% and 1 % data usage. The dataset-specific results can be found in Figure C.I (within Appendix
4.2 ABLATION STUDIES OF AUTOVP
We designed a range of model architectures as testheds for examining the performance of Auto various components. Our comparisons of these VP architectures included 1) the utilization of weight-initialization strategy with FullyMap, 2) the inclusion vs. exclusion of the CLIP text encoder, 3) the presence vs. absence of visual prompts, and 4) the frequency analysis of the learned VP.
Weight Initialization of FullyMap with CLIP When CLIP was used as the pre-trained model, the Map output transformation exhibited significantly inferior performance FreqMap and IterMap (Figure 3). This is because Mrp and IterMap can leverage CLIP's zero-shot property with the semantic meanings random state. As a result, FullyMap tends to perform poorly in the hyper-parameter tuning process, yet may achieve higher accuracy after completing 200 epochs of training. In Figure 3 for example, Auto suggests that the optimal output transformation for Flowers 102 with CLIP is IterMap; but in reality, FullyMap achieves better performance after training for 200 epochs (87.3%, as against 78.8% for IterMap).
the labels, whereas the fully connected layer needs to learn the mapping from a
7

---

Published as a conference paper at ICLR 2024
Using CLIP as the Pre-trained
(a) AutoVP LP
(b) AutoVP -NonVV
CLJP Zeto- Shot Accuracy (%)
Testing Accurayy (7)
M.M--2
AutoV?
Accuracy Gain (5)
60
60
5008
-00
95.2 96.1 94.1 92.9
CLE-A
40
40
x2.2
79.0 1866
76.2
20-
Gras
20-
CIARR
STARIO
40
USC
Teater
Feabir
20
0.4
CIFAR10
CIFARI0-C
Coufidence Score
Coufidence Score
Figure 4: Models Robustness with CLIP. The comparison of accuracy drop on the CIFARIO-C dataset across Auto VP, VP, CLIP-VP and LP..
Figure 5: Accuracy Gains with CLIP. The right side of the chart indicates higher out-of-distribution (OOD) extent, accompanied by larger gain values. Conversely, the left side shows lower gain values.
To address the bottleneck in hyperparameter tuning caused by FullyMap, introduced weight initial- ization (WI). This allows FullyMap to initialize with a more informative mapping based on the seman- tie meaning of the class names. As mentioned in Section 33] Auto FullyMap consists of linear layer, which can be characterized as L, = w- +bb where the weight w is a K, by K,T real matrix, K, denotes the number of target classes, and T is the number of templates used in the CLIP text encoder. The weight initialization is to assign the diagonal of w to be 1, and the rest of it is set to O, resulting in w = (Ix, 0). where K, is an identity matrix of size K,, and is K, x (T 1) ing the connection between each target class class _ame and its corresponding text prompts ("This a photo of [class _aa__assssa_____________________ wwm.mammm"""......... I"') In Figure A. (within Appendix [A.2), we represent this concept vi- sually. As shown in Figure3] our experimental results demonstrate that when utilizing CLIP as the pre- trained model, FullyMap with proper WI (hatched bar) can also outperform other mapping approaches.
zero matrix, indicat-
Impact of the Non-inclusion of Text Encoder in CLIP. When replicating the experimental setting as shown in Table and establishing a direct connection between the fully connected output mapping layer and the CLIP image encoder without incorporating the text encoder, there was substantial decrease in average accuracy: to 69.0% (see Figure E. Appendix [.. Dataset-specific accuracy drops were particularly prominent in Flowers 102, Food101, and OxfordIITPPet. These outcomes suggest that label semantics play a crucial role in those datasets.
The Impact of Visual Prompts. We also investigated the effects on the overall performance of removing the module of visual prompts from the Vu pipeline while retaining the pre-trained model and putput-mapping modules. When the ResNet18 model was used, leaving the visual prompts in yielded better performance than omitting them in just three out of 12 cases: i.e., the SVHN GTSRB, and ISIC datasets (Figure C.3 (b), Appendix C.3) For the remaining datasets, the inclusion of visual prompts actually led to a decline in performance. This suggests that when relatively small source model is used for VP, a significant improvement in accuracy of the sort reported in Table C can primarily be attributed to fully connected layer mapping, and visual prompts may be perceived as noise. On the other hand, when the CLIP model was used, most of the datasets had positive gain values (Figure 5 (b)), indicating improved performance, when visual prompts were included. This suggests that CLIP is more suitable for VP tasks than ResNet18 is.
Frequency Analysis of the Learned Visual Prompts. In Appendix D.I we also conducted an analysis from frequency perspective (Brigham 1988) to study the generalization of visual prompt patterns. The results highlighted the effectiveness of prompting with CLIP, hamesssing freuureyy features that generalize to the target domain.
DISCUSSIONS
Tuning Selection. AutoVP provides joint optimization of its multiple configurations and selects different parameters according to its target tasks. In terms of output label mapping, FullyMap exhibits superior performance in vision models, but IterMap or FreqMap appear to enhance the performance of text-image models like CLIP. In this context, weight initialization with FullyMap plays an important role in CLIP, making this option one of the more frequently chosen output- mutput-mappin strategies (Table 2). We also observed that novel designs exploiting larger image scales and mapping a larger number
8

---

Published as a conference paper at ICLR 2024
of source classes tended to yield enhanced performance. More information on selection preferences during hyperparameter tuning can be found in Appendix D.3
AutoVP Robustness. We trained PP on CIFARIO and evaluated its robustness on the corrupted dataset CIFARIO-C, which consists of 18 types of filters or noise. As shown in Figure 4 Auto maintained greater robustness than ILM-VP, CLIP-VP, and LP. Its loss of accuracy was relatively small: suggesting that Au exhibits a lower degree of overfitting to the training data and possesses higher ability to resist the impact of noise than the other baselines do.
Performance Evaluation on ID/OOD Downstream Tasks. We evaluate the out-of-distribution (OOD) extent of each dataset relative to the pre-trained CLIP by considering the average confidence score (Guo et al.] 2017) and the CLIP zero-shot inference. The accuracy gains achieved through VP (Figure 5) were computed the difference in accuracy between VP and LP or non-VP approaches (I.e. visual prompts were removed and output mapping retained). We observed that the datasets that were more in- ni-istribution (ID), with higher confidence scores and higher zero-shot accuracy, exhibited smaller accuracy gains from VP. Conversely, datasets that were more OOD, characterized by lower confidence scores and lower zero-shot accuracy, had their accuracy improved more through Vu.
We also evaluated accuracy gains with ResNet18 pre-trained on ImageNet- 1 (Russakovsky et al. 2015) (Figure C.3 Appendix C.3) and, to assess domain differences between ImageNet- IK and other downstream datasets, we calculated the FID score (Heusel et al. 2017). The results were consistent with the cases using CLIP. In conclusion, Au is suitable for datasets that exhibit more OOD characteristics than the source domain dataset.
LIMITATIONS
This work is subject to some limitations. First, our optimization process did not include certain hyper- parameters, notably learning rate and weight decay. This omission stemmed from our primary focus on identifying the best configurations for VP training, and because including such hyperparameters would have greatly increased execution workload. In addition, when it comes choosing the best pre-trained model to fine-tune on the target dataset. You et al. (2021) also argued that, in general, the sophisticated fine-tuning techniques (c.g., regularization) would not change the ranking pre-trained models in downstream tasks. Nonetheless, we conducted supplementary tuning experiments encom- passing various learning rates and weight decays, the results of which can be found Table [E.3] (within Appendix E.3 In practice, we suggest enabling the tuning of learning rates, weight decay, and other model-specific parameters after the initial hyperparameter tuning phase of Auto The tuning of fundamental hyperparameters could potentially be accelerated with recent advancements in utilizing generalization metrics to identify optimal hyperparameter configurations (Zhou et al. 2023a b), subject to be explored in future research.
Another limitation pertains to the scope of Vu,o which is oriented toward classification tasks. However, we have extended its application to segmentation and detection tasks, detailed in Appendix E.2 Furthermore, recent studies have demonstrated the success of visual prompts in generative tasks (Ramesh al. 2021 2022 Liu & Chilton| 2022 Bar et al. 2022 Sohn al. 2023).
Nevertheless, expanding support for generative tasks will require accommodation of their distinctive requirements, c.g., via the integration of a suitable pre-trained generative model. such GANs (Goodfellow et al. (2014), VAEs (Kingma & Welling 2013), or diffusion models (Ho et al. 2020). along with tailored prompt design. Certainly, our results imply that there are many avenues for VP research that merit further exploration.
7 CONCLUSION
This paper has introduced Vu, an end-to-end framework that automatically selects the optimal VP configuration for a downstream dataset. Au demonstrates superior performance over other state-of-the-art VP methods and transfers learning baselines in both standard and sample-reduced fine-tuning settings. This research has also yielded important insights into optimal VP configurations, the effects of downstream data characteristies on VP, and how robustness against image corruption might be improved. In short, we believe Auto is an efficient and expandable tool, and perhaps more importantly, useful benchmark that will accelerate the development of VP research and technology.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The research described in this paper was conducted in the JC STEM Lab of Intelligent Design Automation, which is funded by The Hong Kong Jockey Club Charities Trust.
REFERENCES
Huzaifa Arif, Alex Gittens, and Pin-Yo Chen. Reprogrammable-PL: Improving utility-privace tradeoff federated learning via model reprogramming. In First IEEE Conference on Secure and Trustworthy Machine Learning, 2023.
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale scale models. arXiv preprint arXiv:2203. 1(3)4, 2022.
Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. Advances Neural Information Processing Systems, 35:25005-2501 2022.
Lukas Bossard, Matthicu Guillaumin, and Luc Van Gool. Food-101 mining discriminative compo- nents with random forests. In European Conference on Computer Vision, 2014.
Oran Brigham. The fast Fourier transform and its applications. Prentice-Hall Inc., 1988.
Aochuan Chen, Peter Lorenz, Yuguang Yao, Pin- Yu Chen, and Sijia Liu. Visual prompting for adversarial robustness. ICASSP 2023-2023 IEFE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 1-5. IEEE, 2023a.
Aochuan Chen, Yuguang Yao, Pin-Yu Chen, Yihua Zhang, and Sijia Liu. Understanding and improving visual prompting: label-mapping perspective. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 19133-19143, June 2023b.
Liang-Chieh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv:1706.0533 2017.
Pin-Yu Chen. Model reprogramming: Resource-efficient cross-domain machine learning. arXiv
preprint arXiv:2202. 20020 2022.
Gong Cheng. Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 185 1883, 2017.
Gordon Christie, Neil Fendley, James Wilson, and Ryan Mukherjee. Functional map of the world. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018.
M. Cimpoi, S. Maji. I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of the IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2014.
Noel Codella, Veronica Rotemberg, Philipp Tschandl, M Emre Celebi, Stephen Dusza, David Gutman, Brian Helba, Aadi Kalloo, Konstantinos Liopyris, Michael Marchetti, et al. Skin lesion analysis toward melanoma detection 2018: A challenge hosted by the international skin imaging collaboration (isic). arXiv preprint arXiv:1902.03366 2019.
Gamaleldin F. Elsayed, Ian Goodfellow, and Jascha Sohl-Dickstein. Adversarial reprogramming of neural networks. In International Conference on Learning Representations, 2019.
Mark Everingham, SM Ali Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111:98 136. 2015.
Tianyu Gao, Adam Fisch, and Dangi Chen. Making pre-trained language models better few-shot learners. In Association for Computational Linguistics (ACL), 2021.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances neural information processing systems, 27, 2014.
10

---

Published as a conference paper at ICLR 2024
Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Weinberger. On calibration of modern neural networks. In International conference on machine learning, PP. 1321-1330. PMLR, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851 2020.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The german traffic sign detection benchmark. In The 2013 International Joint Conference on Neural Networks (IJCNN), pp. 1 1-8, 2013.
Lei Hsiung, Yung-Chen Tang, Pin-Ye Yu Chen, and Tsung- Yi Ho. NCTV: Neural Clamping Toolkit and Visualization for Neural Network Calibration. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37 (13), pp. 16446-16448, Sep. 2023.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Diederik Kingma and Max Welling. arXiv:1312.6114, 2013.
Auto-encoding variational bayes. arXiv preprint
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pp. 554-561, 2013.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, Toronto, Ontario, 2009.
Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-efficien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 3045-3059, Online and Punta Cana, Dominican Republic, 11 2021. Association for Computational Linguistics.
Liam Li, Kevin Jamieson, Afshin Rostamizadeh, Ekaterina Gonina, Moritz Hardt, Benjamin Recht, and Ameet Talwalkar. Massively parallel hyperparameter tuning. arXiv preprint arXiv:1810.05934, 5, 2018.
Ning Liao, Bowen Shi, Min Cao, Xiaopeng Zhang, Qi Tian, and Junchi Yan. Rethinking visual prompt learning masked visual token modeling, arXiv preprint #xXiv2303300499, 2023.
Richard Liaw, Fric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and lon Stoica, Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05113 2018.
Vivian Liu and Lydia B Chilton. Design guidelines for prompt engineering text-to-image generative models. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1-23, 2022.
11

---

Published as a conference paper at ICLR 2024
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision. pp. 10012-10022, 2021.
Jochem Loedeman, Maarten C Stol, Tengda Han, and Yuki M Asano. Prompt generation networks for efficient adaptation of frozen vision transformers. arXiv preprint arXiv:2210006666, 2022.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pp. 181-196, 2018.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, et al. Simple open-vocabulary object detection. In European Conference on Computer Vision, pp. 728-755. Springer, 2022.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Ng. Reading digits in natural images with unsupervised feature learning. NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
M-E. Nilsback and A. Zisserman. Automated flower classification over large number of classes. In Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008.
Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTaek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, and Kyungwoo Song. Blackvip: Black-box visual prompting for robust transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 24225-24235, June 2023.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. 2012 IEEE conference on computer vision and pattern recognition, pp.
IEEE, 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. sero-shot text-to-image generation. In International Conference on Machine Learning, pp. 8821-8831. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.066 06622.. 1(2):3, 2022.
E. Riba, D. Mishkin, D. Ponsa, E. Rublee, and G. Bradski. Kornia: open source differentiable computer vision library for pytorch. In Winter Conference on Applications of Computer Vision, 2020.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV). 115 (3):211-2522 2015.
Fobo Shi, Peijun Qing, Dong Yang, Nan Wang. Youbo Lei, Haonan Lu, and Xiaodong Lin. Prompt space optimizing few-shot reasoning success with large language models. arXiv preprint arXiv:2306.03799, 2023.
Kihyuk Sohn, Huiwen Chang, José Lezama, Luisa Polania, Han Zhang, Yuan Hao, Irfan Essa, and Lu Jiang. Visual prompt tuning for generative transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19840-19851. 2023.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset 101 human actions classes from videos in the wild. arXiv preprint arXiv: 222 2012.
12

---

Published as a conference paper at ICLR 2024
Hao-Lun Sun, Lei Hsiung, Nandhini Chandramoorthy, Pin-Yu Yu Chen, and Tsung- Yi Ho. NeuralFuse: Learning to Recover the Accuracy of Access-Limited Neural Network Inference Low- Voltage Regimes. arXiv preprint arXiv:2306.16800 2023.
Yung-Chen Tang, Pin-Yu Chen, and Tsung-Yi Yi Ho. Neural clamping: Joint input perturbation and temperature scaling for neural network calibration. arXiv preprint arXiv:2209. 16004, 2022.
Yun-Yun Tsai, Pin- Yu Chen, and Tsung- Yi Ho. Transfer learning without knowing: Reprogramming black-box machine learning models with scarce data and limited resources. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 9614-9624, PMLR, 13-18 Jul 2020.
Yun- Yun Tsai, Chengzhi Mao, and Junfeng Yang. Convolutional visual prompt for robust visual perception. In Advances in Neural Information Processing Systems, volume 36, pp. 27897-27921. Curran Associates, Inc., 2023.
Philipp Tschandl, Cliff Rosendahl, and Harald Kittler. The ham 10000 dataset, large collection of multi-souree dermatoscopie images common pigmented skin lesions. Scientific data, 5(1):1-9, 2018.
Junyang Wu, Xianhang Li, Chen Wei, Huiyu Wang, Alan Yuille, Yuyin Zhou, and Cihang Xie. Unleashing the power of visual prompting at the pixel level. arXiv preprint 10X5::2212.10556 2022.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 IEEE computer society conference on computer vision and pattern recognition, pp. 3485-3492. IEEE, 2010.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1492-1500, 2017.
Ziqing Yang. Zeyang Sha, Michael Backes, and Yang Zhang. From visual prompt learning to zero-shot transfer: Mapping is all you need. arXiv preprint 23t.::3333.052666 2023.
Hao Yen, Pin-Jui Ku, Chao-Han Huck Yang, Hu Hu, Sabato Marco Siniscalchi, Pin-Yu Chen, and Yu Tsao. Neural model reprogramming with similarity based mapping for low-resource spoken command classification. arXiv preprint arXiv:2/ 2021.
Kaichao You, Yong Liu, Jianmin Wang, and Mingsheng Long. LogME: Practical assessment of pre-trained models for transfer learning. In International Conference Machine Learning, pp. 12133-12143. PMLR, 2021.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 168 1-16625,, 2022a
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision, 130(9):2337-234) 2022b.
Yefan Zhou, Tianyu Pang. Keqin Liu, Charles H. Martin, Michael W. Mahoney, and Yaoqing Yang. Temperature balancing, layer- layer-wise weight analysis, and neural network training. In Thirty-eevnnth Conference on Neural Information Processing Systems, 2023a
Yefan Zhou, Yaoqing Yang, Arin Chang, and Michael W. Mahoney. three-regime model of network pruning. In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pp. 42790-42809, 2023b.
13

---

Published as a conference paper at ICLR 2024
APPENDIX
A
Implementation Details of AutoVP
15
Pre-trained Classilier Details
15
Output Lahel Mappings of Au
15
A.3
AutoVP Tuning Process
17
Datasels and Baselines
The Twelve Downstream Datasets
18
B.2 Baselines Details
18
C Additional Experiments with AutoVP
19
Fixed Pre-trained Model vs. Auto Pre-trained Model Selection
19
Data Scalability
20
Downstream Dataset Analysis (IDIOOD vs. Accuracy Gain)
21
D) Analysis of Auto VP Results
22
D.I Prompts Frequency Domain
22
D.2 Output Mapping Analysis
23
D.3 The Preferences in Hyper- Parameter Tuning Selection
24
E
Ablation Studies
24
E.1
The Impact Text Encoder CLIP
24
Visual Prompting Segmentation and Detection Tasks
................................
25
E.3
Exploring Additional Tuning Axes
27
E,4
Improved ILM VP with Tuning Configuration
27
E.S
Comparison of AutoVP and BlackVIP
28
F
Performance and Resource Utilization
28
Comparison AutoVP, Linear Probing, and Full Fine- Tuning
................................
28
Computing Resources
29
14

---

Published as a conference paper at ICLR 2024
A IMPLEMENTATION DETAILS OF AUTOVP
A.1 PRE-TRAINED CLASSIFIER DETAILS
Our Vu framework includes four pre-trained models: ResNet18 (He al. 2016). ResNext101- IG Mahajan et al. 2018), Swin-T (Liu et al.) 2021), and CLIP (Radford et al. 2021). Both ResNet18 and Swin-T models were trained on the ImageNet-IK (Russakovsky et al., 2015) dataset, while ResNeXt101-IG was pre-trained on a large collection of Instagram photos. Additionally, CLIP was trained dataset consisting 400 million pairs of images and corresponding text from the internet.
In the structural differences, the first three models are single-modalee vision models, ResNet is typical and relatively small convolutional neural network with residual blocks. ResNeXt101-IG is deeper residual network that incorporates cardinality, which refers to the size of the set of transformations (Xie et al. 2017). Swin-T a vision transformer that operates by dividing the input image into patches and processing them using the transformer architecture. The last model, CLIP, is lssounlanguuge multi-modal model that calculates the cosine similarity between image embeddings and label text embeddings. The prediction is to select the class with the highest similarity score to the embedding the input image. The prediction flow is illustrated in Eq. 3 where the input image denoted as and K, represents the size of the predictable class set. The token vector of the i-th class label text, obtained from a tokenizer, is denoted as ClsTk; CLIP utilizes the and Text--nnoderr) components extract features from images and text. The resulting image and text embeddings are represented as Temb and temb. respectively. The cosine similarity between any pair of (image, text) embeddings can computed, and the class with the highest cosine similarity considered as the predicted class 3/peed
Lemb (ange-Encoder(x))
t temb; Text-Encoder( ClsTk;), OSi< K,
Lemb temb.
Ypred
arg
OSicKe
Temb tembb
By training with pairs of image and text annotations, CLIP is able to learn correlations between visual and textual information, achieving state-of-the-an zero-shot accuracy.
A.2 OUTPUT LABEL MAPPINGS OF AUTOVP
As mentioned in Section 3 Vu incorporates four output label mappings: frequency mapping (FreqMap), iterative mapping (IterMap), semantic mapping (SemanticMap), and fully connected layer mapping (FullyMap). In the following, provide more details of each mapping algorithm.
Frequency Mapping
Mapping
Semantic Mapping
Embedding classes names
Source Labels
Labels
Mapping spoch
Source
VP Model
Testt Encoder
Target names
Embeddding of target classes
Mapping similar
IV. Fully Connected Layer Mapping
V. Fully Connected Layer Mapping: CLIP with Weight Initialization
Logit
Logit
Pred.
Pred.
class
of Target classes
Target classes
Source classes
al templatesx tanfttartt--llasss
Default Template: This a photo of
Figure A.l: Output Label Mapping. Illustration of four output mapping methods.
15

---

Published as a conference paper at ICLR 2024
A.2.1 FREQMAP (FREQUENCY MAPPING)
In FreqMap. the mapping established between each target class and the most frequently mapped source class. We demonstrate the general many-to-one mapping (multiple source labels mapped to one target label) in Algorithm| [ In lines 6-10, we traverse the training data pairs (#i, yr). First, we pad the image 2; with the current visual prompts to obtaín prompted image zt. Then, we could obtain the prediction fa, (āt) = Ye in the source domain. This gives us a mapping relation from ys to yt and increase the count accordingly. In line 11, obtain the list of source-target id pairs (Saa, Tia), which is sorted by the count matrix (count) in descending order. Accordingly, we start defining the mapping of FreqMap from the most frequently source-target pair. If the source class has not been mapped yet and the mapping count for the target class has not reached the limit m, we establish the mapping relationship M [[aa][]ia] 1 (line 14). We continue this process until all target classes have been mapped to m source classes. Once this condition is met, the mapping assignment completed, and we return the mapping matrix M.
Algorithm 1: FreqMap (6, Dr, m)
Input:
visual prompts 6, source classifier Sa, target dataset D,, and the specified number of source classes mapped each target class
Output: mapping matrix M
# Initialization
Ks, K, < number of source and target classes
M xK mapping matrix
count + R.x.. a zero matrix, records the number of images of each target class being predicted (by fo,) as each source class
done_s +- 0k.
# a boolean vector, records whether the source class has been assigned boolean vector, records whether the number of source classes per target class is equal to m
done_t 0K:
Calculate the Frequency
foreach (Ii, V1)
D, do
& P(n))
generates é. from in and using Eq. I 3/4 fo, (Tx) # the predicted source domain class
count [yd] + count count[yslvvi +
10 end
11 index 1ist + ArgSor((count) get the list of source-target id pairs (Sia, Tia) sorted by count [Sa][[S in descending order
Define the Fap
12
for (Saa, in index_liss list do if not done_t [Ta] and not done_s(Sul) then
13
[(Saa] [Ta) +-
# assign the mapping from Saa (source label) to Tia (target label)
15
done_s(Sia) + True
end
17
Sum
(M[: [aa) = m then
done_t [Ta] <- True # the target class Tia has been mapped to m source classes
end 19
29
if Sum(done_t) = K, then
break # early stop all the target classes has been mapped to m source classes
22
end
23 end
return M
A.2.2 ITERMAP (ITERATIVE MAPPING)
In IterMap, the beginning of each training epoch, the mapping relationship is established using the FreqMap with the current prompted image. This means that the visual prompts updated via the VP model are padded onto the image and participate in the mapping process. Hence, this mapping relationship changes as the visual prompts are trained, which is known as bi-level optimization (Chen al. 2023b).
16

---

FullyMap utilizes linear layer L, = L, + b with weights and bias b to learn the mapping. enabling the transformation of the output source logits I, to target logits I,, As for weight initial- ization (WT), it employed in the case of CL.PP with FullyMap. This technique involves setting the weights between the target labels and their default templates 1, while setting the rest to O. With Published as a conference paper at ICLR 2024
A.2.3 SEMANTICMAP (SEMANTIC MAPPING)
In SemanticMap. source and target classes with semantically similar class names are mapped together. This mapping is accomplished using CLIP's tokenizer Tokenizer() and text encoder Text-Encoder(). We demonstrate the mapping process using the following equation:
ClsTky, Tokenizer( N,
ClsTky y. Tokenizer y
Emby, Text-Encooee (Clsky
Emby, yn Text-Encoder ClsThy
(4)
Emby. Emby
Similarity Emby |II Embyn
y1 - y arg max (Similarilyyg..w) WE
In Eq. let 34+ (0...., K2 - 1} and Di = {0, ..... , 1 be the set of source and target labels, where KjKK, the numbers of source/target classes. For the source label y. E 3 with the classname Ny, and the target label 3/ 34 with the classname N we first utilize the tokenizer to obtain token vectors (ClsTky, and ClsTky, corresponding to N y, and Ny. Then, the text encoder is used to obtain their embeddings (Emby. and Emby.) ). Pair-wise cosine similarity is calculated between each source and target embeddings, and each target label is mapped to the source label with the highest similarity.
A.2.4 FULLYMAP (FULLY CONNECTED LAYER MAPPING)
WI, the linear layer can achieve a favorable initial mapping state, thereby expediting the process of obtaining good mapping relation.
A.3 AUTOVP TUNING PROCESS
In Figure there are three stages involved in the tuning process, while the Visual Prompt component depicted in Figure [ is not involved in the tuning process, it does not contain any hyper-parameters, During the Input Scaling stage, the initial scale of the input image is determined, and users choose whether learn the resize scale during training. In the Pre-trained Classifier stage, users have the option to select from four pre- gre-trained models to serve as the feature extractor. The Output Label Mapping stage offers four mapping methods to choose from. For FreqMap. IterMap, and Map, users can specify the number of source classes that are mapped to single target class.
+LS
ResNet18
FreqMap
0.5
Trainable Resize
ResNex1100-GG
IterMap
1.0
non-Traien non-Trainable Resize
10
1.5
Swin-T
SemanticMap
Number of Source
Input Image
Classes Mapped to Each Targel Class
CLIP
FullyMap
Input Scaling
Classifier
Oulput Label Mapping
Figure A.2: Hyper-Parameter Tuning Selection. Illustration of the end-to-end hyper-parameter tuning process in Auto with total of 222 possible configurations.
17

---

Published as a conference paper at ICLR 2024
B DATASETS AND BASELINES
THE TWELVE DOWNSTREAM DATASETS
To assess the efficacy of the proposed VP, selected the following twelve datasets for experiments. The detailed information is shown in B.1
CIFARIO & CIFARI00 (Krizhevsky & Hinton 2009): ): The datasets consist labeled subsets of the 80 million tiny images dataset, which composed of 32x32 color images.
ISIC (Codella al. 2019; Tschandl al. 2018): The International Skin Imaging Collaboration
(ISIC) developed an international repository of dermoscopic images known as the ISIC Archive, The images of the datasets were acquired using different devices at several medical centers worldwide.
SVHN (Neteer al. 201T): real-world image dataset of street view house numbers.
GTSRB (Houben et al. 2013): The German Traffic Sign Benchmark (GTSB) is a multi-class, single-image classification challenge that was conducted at the International Joint Conference on
Neural Networks (ICNN) in 2011.
Flowers102 NNllsback & Zisserman 2008): The 102 categorics flowers dataset consists of com- monly occurring flowers in the United Kingdom.
DTD Cimpoi et al. 2014): The Describable Textures Dataset (DTD) is collection of textural images that have been annotated with a series of human-centri attributes.
Food101 (Bossard et al. 2014):
The food dataset consists of 101 different classes, with a total of
101,000 images.
EuroSAT (Helber al.
2019) The Sentinel-2 satellite images dataset for land use and land cover
classification.
OxfordUlIPPet (Pets) Parrkh et al. 2012): The dataset includes diverse breeds of cats and dogs, with images exhibiting variations in scale, pose, and lighting conditions.
UCF101 (Soomro et al., 2012): The action recogmition dataset consists of realistic action videos that have been collected from YouTube.
FMoW (Christie et al. 2018): The dataset contains satellite images that are used for sites and land use classification.
Table B.1: Dataset Setting
Dataset
Class Number
Training Set Size
Testing Set Size
Batch Size
SVHN
10
73,257
26,032
128
EuroSAT
10
13,500
8,100
128
Flowers 102
102
4,093
2,463
64
CIFARI00
100
50,000
10,000
128
UCFI01
101
7,639
3,783
128
DTD
47
2,820
1,692
FMoW
62
76,863
22,108
128
GTSRB
43
26,640
12,630
128
CIFAR10
10
50,000
10,000
128
10od101
101
75.750
25,250
128
OxfordMIIPPet
37
3,680
3,669
128
ISIC
7
4,990
555
128
B.2 BASELINES DETAILS
The reported accuracy of the baselines (ILM-VI LP by Chen et al. (2023b). CLIP-VP and LP by Bahng al. (2022) in Section is obtained from the results documented in the respective papers. For some datasets (such as ISIC, FMoW, and GTSRB), the authors did not include them in their paper; in this regard, we follow the corresponding experimental settings obtain baseline accuracy.
18

---

Published as a conference paper at ICLR 2024
C ADDITIONAL EXPERIMENTS WITH AUTOVP
C.I FIXED PRE-TRAINED MODEL vs. AUTO PRE-TRAINED MODEL. SELECTION
We present additional experimental results highlight the effectiveness of AutoVP. In addition to the comparisons with CLIP and VP baselines discussed in Section 4.1 we further evaluate the performance using ResNet18 8 the pre-trained model and explore a scenario without any pre-trained model restrictions. The results using R are presented in Table C.I while the results for the unrestricted scenario are provided in Table C.2 These comparisons consistently demonstrate that Au outperforms previous approaches, including LP and the state-of-the- VP methods.
Table C.I: VP with ResNet18. Comparison of VP test accuracy (%) using ResNet18 8 as the pre-trained model on 12 datasets.
Dataset
AutoVP Setting
AutoVP
ILM--V
I.P
SVHN
FullyMap. p 48
83.74 = 0.45
75.2
65.0
CIFARIO
FullyMap, p 48
87.81 0.17
65.5
85.9
Flowers10s
Ful p 16
85.40 + 1.89
27.9
88.0
Food101
FullyMap, p 16
54.15 = 3.53
14.8
50.6
UCF101
FullyMap, p 16
55.86 + 1.81
23.9
63.2
OxfordllTPPet
FullyMap. p 16
82.65 = 0.84
65.4
87.2
CIFARI00)
FullyMap. 16
63.67 + 3.48
24.8
63.3
EuroSAT
FullyMap, p 48
93.01 0.15
85.2
93.8
DTD
FullyMap, p 16
54.82 = 1.14
35.3
60.0
ISIC
FullyMap. 16
67.44 + 1.22
57.5
68.6
FMoW
FullyMap, p 16
30.17 + 0.06
14.8
28.4
GTSRB
FullyMap, p
81.52 1.21
52.0
77.4
Average Accuracy
70.02
45.2
69.3
Table C.2: PutoVP with source model selection. This table displays the best tuning setting without any restriction on the choice pre-trained model, and shows the test accuracy (%) of Auto and the LP baseline of the chosen model across 12 datasets.
Dataset
AutoVP Setting
AutoVP
LP
SVHN
CLIP, FullyMap. p 51
92.86 0.18
65.40
CIFARIO
ResNeX:101-IG, FullyMap,
95.89 0.07
93.89
Flowers102
Swin-T, FullyMa,, p
93.48 0.52
95.75
Food101
CLIP, FreqMap-1. p 16
82.28 +: 0.09
84.60
UCF101
Swin-T, FullyMap. 16
72.96 + 0.26
75.96
Oxford
Swin-T. FullyMap. 16
90.20 +: 0.55
93.04
CIFARI00
ResNeXU11-IG, FullyMap. 48
79.76 + 0.47
76.09
EuroSAT
Swin-T, FullyMap,
95.98 0.02
95.50
Swin-T, MaplyMap. p
69.25
71.49
ISIC
Swin-T, FullyMap, p
71.66 1.45
72.22
FMoW
Swin-T, FullyMap, 48
39.79 +: 0.83
32.73
GTSRB
Swin-T, FullyMap. 55
88.10 10 + 2.11
74.97
Average Accuracy
81.02
77.64
19

---

Published as a conference paper at ICLR 2024
C.2 DATA SCALABILITY
In the data scalability experiments, Figure C.I illustrates the performance of Au and LP on each dataset under various data usage proportions. The corresponding settings can be found in Table When the chosen pre-trained model is fixed to CLIP, Vu outperformed LP in scenarios with limited data on most of the datasets. In some datasets, such as SVHN, GTSRB, and EuroSAT, Au performs better than LP for all proportions of data. Besides, in OxfordIIITPet, CIFARI00, DTD, Food101, UCF101, and Flowers Au also obtains promising performance compared to LP on small amount of data. This suggests that VP will be more suitable and effective solution than LP, especially when training data limited.
C,3
AutoVP Topl Accurccy
Accurecy
SVIIN
EuroSAT
Oxfordil'TPet
90
80
90
60
60
70
40
20
11025 50
(%)
11025
100 (%)
11025 50
100
11025
100 (%)
CIFARI00
DTD
Food101
UCF101
85
80
60
60
40
70
40
20
65
20
11025
50
100
11025
100 (%)
11025 50
100
11025
100 (%)
Flowers102
CIFAR10
ISIC
FMoW
100
80
70
92
60
90
50
20
11025
110.55
100 (%)
50
(%)
100 (%)
Figure C.1: Data Scalability. The charts present the accuracy of Au and LP with varying percentages of data usage: 100%, 50%, 25%, 10%, and 1%.
Table C.3: Data Scalability Settings. The table shows the settings for data scalability experiments with data usage ranging from 100% to 1%. The source pre- trained model is fixed to CLIP. The notation "Mapping-m"" represents mapping source classes each target class.
Dataset
100%
50%
25%
10%
1%
SVHN
FullyMap, p = 53
FullyMap, p = 51
FullyMap, p 49
FullyMap, p
FullyMap, p 80
CIFAR1O
IteMap-1. = 22
FreqMap-1. p 16
FreqMap-S. p
ItrMaa--5.
IteMaa--, p
Flowers102
FullyMap. 16
FreqMap-1. 16
FreqMap-1. 16
IterMap- 1. 16
FullyMap, p 16
Food101
PregMap-1, p 16
IterMap-5, p
16
IterMap-1, 16
IterMap11, 16
FullyMap, p 16
UCF101
FullyMap. p 17
FullyMap. p 16
IterMap-1, p=
FullyMap. p
FullyMap, p
OxfordIITPPt
FregMap-10 10, 16
FreqMap 16
FreqMap p 16
FullyMap.
FullyMap, p 16
CIFAR100
FullyMap. 30
FullyMap. = 26
IterMap-1. = 16
FroqMap-1. p
FullyMap, p 16
FallyMap, p 16
FallyMap, p 16
FullyMap, p
FullyMup.
Fully p
DTD
FullyMap. p 18
IterMap- p 16
IterMap-S,
FullyMap.
FullyMap, p 16
ISIC
1trMap-1. 16
FreqMap-1 1, 16
FullyMap, 80
FreqMap-S. 48
FullyMap. 80
FMoW
FullyMap. p 16
FullyMap, p 16
FullyMap, p=
FregMap-10.
FullyMap. p 17
GTSRB
FullyMap. p 80
FullyMap. p 16
FallyMap, p
FullyMap.
FullyMap, p 80
20

---

Published as a conference paper at ICLR 2024
C.3 DOWNSTREAM DATASET ANALYSIS (ID/OOD vs. ACCURACY GAIN)
To further understand how the downstream dataset distribution influences the performance of visual prompting. We conduct experiments to observe the relation between accuracy gain and dataset characteristics. When using CLIP as the pre-trained model, we define confidence score, obtained by averaging the maximum softmax probability predictions across the entire training set, as an indicator of dataset characteristics. For other vision pre-trained models, the FID score is used to measure the dissimilarity between the downstream dataset and the ImageNet-1K dataset, i.e. the degree of out-of ditro--dittiibtionnn (OOD). We present the Confidence/FIDD scores of each dataset in Figure C.2. providing insights into their OOD characteristics. Furthermore, Figure C.3 demonstrates the accuracy gain for each dataset when using R 8 the pre-trained model. The experimental results show that when using VP, datasets with a higher degree of OOD tend to benefit from more accuracy gains.
Confidence Score
FID Score
1.0
250
0.8
211
200
0.6
15A.5
150
100.
0.2
100
95.
000
0.0
50
Fuallu
Urtiul
Howenatt
CIPARa
LTTD
Iac
CITRU
SWIN
UeTlus
DTD
Punditi
Phlow
Rowerste
BandAr
bsc SVIN orsta
Figure C.2: Confidence Score and FID Score. We sort the datasets by the degree of out-of- distribution (OOD), where the one closer to the left indicates higher similarity (in-distribution, ID) to the training data of the pre-trained model, while the one closer to the right indicates greater dissimilarity (OOD).
(a) AutoVP LP
(b) AutoVP Non-VP
20
20
Accuray Gain (%)
XNTIN
10
10
GENAB
CIFAR10 RMW
CIFARIM
Bowesat22
0
Fond
CIFASAM
BS
Boowesiie
-10
-10
100
150
200
250
100
150
200
250
FID Score
FID Score
Figure C.3: Accuracy Gains with Resnet18, The gains are calculated by taking the difference between the performance of Vu and LP or Non-Ve VP scenario.
Table 2 has encompassed the prevalent 12 datasets in VP research. Furthermore, to enable a more comprehensive comparison within wider spectrum of VP research, we have included additional datasets-SUN33977
et al., 2010), RESISC(Cheng et al. 2017), CLEVR[Jhnson et al. 2017). and StanfordCar( Krauee et al. 2013) -with their respective results available in Table C.4 Our findings consistently demonstrate Auto superior performance compared to ILM and CLIP-VP across the most of datasets. Especially notable is the substantial 15% increase in accuracy observed in the out-of-distributie the on-distribution (ID) datasets, SUN397 and StanfordCar, showcased lower performance than LP, aligning with the trend of accuracy gain illustrated in Figure 5 in Section
(OOD) dataset, CLEVR. when compared to linear probing (LP). However,
21

---

Published as a conference paper at ICLR 2024
Table C.4: Performance on Additional Datasets. The
testing accuracy (%) for four additional datasets com- monly found in VP research. The figure on the right side shows the aceuracy gains.
Auto-VP LP
CLIP CLIP Zer Shot Accuracy (%)
15-
CLEVR
Accuray Gain (%)
Dataset
Vu Setting
AutoVP
LP
ILM
CLIP-VP
SUN397
FullyMap. p=
65.4
70.9
61.2
60.5
RESISC
30
StanfordCar
PullyMap, p= 16
61.8
77.6
57.6
56.2
SUN397
RESISC
FullyMap.
88.5
91.7
86.6
84.5
StanfordCar
CLEVR
FullyMap.P p=
68.0
83.1
81.4
16
Confidence Score
D ANALYSIS OF AUTOVP RESULTS
D.1 PROMPTS FREQUENCY DOMAIN
We have analyzed the learned prompts using the best setting selected from Auto and have represented them in the frequency domain through Fast Fourier transformation (Brigham 1988). In FigureD.1 a), the prompting result of SVHN dataset with CLIP notably distinct and achieves the highest testing accuracy among all the pre-trained models. In the frequency analysis (Figure (.), the prompts are concentrated in the low-frequency domain (at the center of the plot), with CLIP displaying the most distinct structure and significantly larger magnitudes compared to the others. These results validate the efficient learning of prompts with CLIP, harnessing uwwfreeuency features that generalize to the target domain.
AutoVP with ResNet18 Testing Acc-83.7%
AutoVP with ResNeX1101-IG Testing Acc-85.6%
Au with Swin-T Testing Acc=90.7%
AutoVP with CLI
Testing 9cc-92.9%
92.9%
25-
27-
27-
20-
50
50-
130
100
135
125
135
150
150
150
150
190
:00
(a) Learned Prompts with Different Models
AutoVP with ResNet18
AutoVP with ResNeXt101-IG
AutoVP with Swin-T
AutoVP with CLIP
75
100
100
100
130
125
200
200
200
230
100
1.0
200
60
100
152
200
60
130
150
200
50
100
150
(b) Prompts in Frequency Domain
Figure D.I: Prompting Analysis of SVHN Using Various Pre-trained Models. (a) Frame-shape prompts learned with the best settings selected from Au for given pre-trained model. (b) Prompts in the frequency domain by Fast Fourier transformation.
22

---

Published as a conference paper at ICLR 2024
D.2 OUTPUT MAPPING ANALYSIS
(a) The Weight Distribution bumpy FullyMap
(b) The Weight Distribution of scaly in FullyMap
Distribution tullyftep
Distribution FullyMap
upoee
conaticIs
cate
ruby
CONTTRII SpAuTA
Figure D.2: The Weight Distribution of Mup in DTD. The top 5 source labels exhibiting the highest weight values within the FullyMap pertain (a) the label bumpy and (b) the label scaly.
Figure D.2 illustrates that FullyMap can be interpreted as weighted combination of multiple source labels, where some readabble features may exhibit similarity. For instance, in Figure D.2 a), humpy shows similarities with custard apple, disk brake, and pineapple, while in Figure D.2 b), scaly shares similar features with boa constrictor, coho, and common iguana.
banded
braided
Target Label
(a)
DTD with swin-t & FullyMap, Acc: 69.96%
fgg
gondole
apple
Inrikisha
COCK pot
Source Label
CLUB
(b) DTD with swin-t & IterMap-1, Acc: 40.77%
web site
acom
knot
butbble
custard agple
crossword puzzle
spider web
Source Label
Figure D.3: The Correspondence Between Output Mapping Labels in DTD, Each column represents the mapping between the target label and its respective top-1 source label in FullyMap. (a) FullyMap (top-l class having the largest weight) with Swin-T (second row). (b) IterMap-] with Swin-T (third row).
Furthermore, when comparing FullyMap and IterMap, a sigmificant accuracy gap observed, with Fup achieving 99.96% and IterMap-1 only achieving 40.77%. However, in Figure D.3 IterMap has mapped to some classes that are indeed very close to the target. For instance, in Figure D.3 (b),
braided maps to knot, bubbly maps to bubble, and cobwebbed maps to spider web. This demonstrates that mere combination source labels insufficient for achieving better performance; the weighting in the combination plays crucial role, which is precisely what FullyMap accomplishes.
23

---

Published as a conference paper at ICLR 2024
D.3
THE PREFERENCES HYPER- SHEER-PRRMMEER TUNING SELECTION
Pre-trained
80
50-
Accumey
ReaNet18
BeNNoXIOL Swiss Pre truiteddddd. Models
CLIP
0.5
1.0
1.3
RedNett
BesNeX0:01.0
Swin T Pre-traized Models
CLP
Initaal Scale
Figure D.4: Average Tuning Results. The three charts display the average few-epoch tuning accuracy for different selected conditions.
Figure D.4 illustrates the average tuning results for all datasets across different settings, including image scale, mapping methods, pre-trained models, and so on (see Figure A.2). In Figure D.4(a). among the vision models (ResNet18 88, ResNeXt-IG, Swin-T), the FullyMap demonstrates superior performance compared to the other methods. However, for CLIP, vision-languee FullyMap only shows a slight advantage over the others. Furthermore, Figure D.4(b) indicates that larger initial scales, such 1.0 1.5 (yielding square images with width of 128 x 1.0 or 128 x 1.5), generally lead to better results when using Map, IterMap, or FullyMap. Last, since both Map and IterMap configure the number (m) of source labels mapped to each target class, we found that increasing this count generally improves accuracy for the three vision models (see Figure FigureD..4jc)) DD4 However, for CLIP, mapping five source labels appears to be the optimal choice based the average tuning results.
model, the
E
ABLATION STUDIES
E.1 THE IMPACT OF TEXT ENCODER IN CLIP
Figure illustrates the impact on accuracy when incorporating or excluding the CLIP text encoder. On average, this configuration results in significant decrease in accuracy of approximately 12%, highlighting the crucial role of the text encoder in VP with CLIP.
CLIP Variation Performance
0411
-0.91
46.35
Testing Accuracy (%)
80
2.16
5.08
-30,05
-1051
5.78
60
40
AuteV/P FullyMi Aug: 20.75
Amp
SVHN
EuroSAT
Floweretop
CIFARiOD
UCF10
DTD
FMoW
GTSRB
CIFARIO
Foodtor
Pet
ISIC
Figure E.1: The Accuracy Difference of Auto Variations. The chart shows the difference in accuracy between AutoVP variations with and without the CL.IP text encoder. The numbers above the bars indicate the accuracy difference.
24

---

Published as a conference paper at ICLR 2024
E.2
VISUAL PROMPTING IN SEGMENTATION AND DETECTION TASKS
(a)
Segmentation
(b) Detection
Trainable
Pre- Trainedd Model
Classifier
Pro-Trained Model
Box Head
Conwaa 21)
Mulliyerr Perouptron
Com2201200,
(MLF)
(a-1) Linear Probing
(b-1) Linear Probing
Pre-
FullyMap Camoh121 3)
Pre-Trained
Model
Model
Visual Prompts
Visual Prompts
(a-2) AutoVP with FullyMap
(b-2) AutoVP without Mapping
Figure E.2: LP and Auto VP in Segmentation and Detection Tasks.
Although AutoVP primarily focuses classification tasks, aimed delve deeper into various vision tasks by incorporating models designed for different purposes. For segmentation tasks, we employ DeepLabV3 (Chen al. 2017) the pre-trained backbone, and for detection tasks, utilize OWL-ViT (Minderer et al. 2022). To evaluate the capability of both in-distribution (ID) and out-of-distribution (OOD) datasets, chose ISIC as the OOD dataset for both tasks. For datasets, we use Pets in the segmentation task and VOC (Everingham et al. 2013) in the detection task.
Our VP segmentation framework, depicted in Figure (a), integrates FullyMap after the pre- trained model to facilitate pixel-wise classification using a custom class number. In contrast, the linear probing approach modifies the final 2D convolutional layer, and the results are depicted in Table E.
Table E.1: Performance on Segmentation Datasets. The IoU and pixel accuracy (%) of linear probing (LP) and AutoVP.
Dataset
LP
AutoVP
Pets
IoU: 0.83, Pixel: 90.7%
IoU 0.77, Pixel : 86.9%
ISIC
IoU 0.64, Pixel: 78.1% IoU 0.81, Pixel 89.5%
We evaluated segmentation performance using two metries: IoU (Intersection over Union) score and pixel-wise classification accuracy. Au exhibited superior performance on both metrics on the ISIC dataset. Additionally, segmentation examples highlighted that predictions align more accurately with the ground truth mask when the prompt space is larger (see Figure E.3). However, in the ID dataset (Pets), VP performance was inferior LP. This aligns with our findings in the classification task, where OOD datasets derived greater benefits from visual prompts.
25

---

Published as a conference paper at ICLR 2024
(a) Linear Probing 0.64, 78.1%
(b) AutoVP (p=16) loU: 0.72, Pixel:
(c) AutoVP (p=48)
lol: 0.81, Pixel:
Image
Predicted Mask
Ground Truth
Figure E.3: ISIC Segmentation. Performance of LP and Au with different prompt sizes (p). It's important to note that the performance calculation involves only the target image itself without the prompts, ensuring fair comparison to LP.. Specifically, the predicted mask is initially cropped to the region of the original target image and then resized to match the dimensions used in LP.
In the case of the detection task, as illustrated Figure E.2 (b), the LP approach fine-tunes on the box predictor head, whereas the VP method incorporates trainable visual prompts. To achieve consistent box output, we have omitted the inclusion of output mapping. The results are shown Table Similar segmentation, the OOD dataset, ISIC, exhibits better performance than LP with larger prompt size.
AutoVP (p=16)
(b) AutoVP (p=48)
Table E.2: Performance on Detection Datasets. The loU scores of linear probing (LP) and AutoVP.
with Prompts
Dataset
Zero-shot
Finetune box head
VP
IoU: 0.78
IoU: 0.29
IoU: 0.81
IoU: 0.81
VOC
IoU: 0.75
IoU: 0.76
IoU: 0.73
Prediction
ISIC
IoU: 0.52
IoU: 0.70
IoU: 0.80
Ground Truth
Predicted Box
26

---

Published as a conference paper at ICLR 2024
E.3 EXPLORING ADDITIONAL TUNING AXES
In Au,oVP, we set the learning rate (LR) for CLIP to be 40 and employed the SGD optimizer with momentum of 0.9 and a weight decay (WD) of O. In this section, have undertaken supplementary tuning options within the Auto framework. Specifically, we introduce additional choices for the LR, selecting from 35, 40, and 45 (for CLIP), as well as WD, choosing from O, 10 5 and 10 10-10, The outcomes presented in Table E.3 showcase a 0.6% enhancement in accuracy with the additional tuning options. It is important note that the execution workload dramatically increases, will involve exploring times more additional combinations in the tuning process.
Table E.3: Hyper-Parameee Tuning for Learning Rate (LR) and Weight Decay (WD). The table displays the optimal tuning configurations and corresponding testing accuracy for both scenarios with and without additional LR and WD tuning options for Auto VP Flowers102. The pre-trained model used is CLIP. The highest accuracy is marked in bold.
Setting
Tuning Selection
Testing Accuracy (%)
FullyMap, p 16
Vu
LR = 40
90.4
w/o LR & WD tuning
WD 0
FullyMap. p 16
Pu
LR 45
91.0
w/ LR & WD tuning
WD 10-10
E.4 IMPROVED ILM-VP WITH TUNING CONFIGURATION
In order to establish an equitable comparison with AutoVP, we undertake a comprehensive hyper- parameter tuning process for ILM-VP. We investigate tuning options that span 1, 2, 5, and 10 source classes for output mapping, as well as prompt sizes of 10, 20, 30, 40, and 50. For ILM-VP without tuning, default settings maintained, with the mapping number set to and the prompt size to 30.
The tuning procedure applied to several datasets, and the results are shown in Table E.4 Although hyper-paramett tuning in ILM- VP leads to an improvement in accuracy, it remains unable to surpass the performance exhibited by VT.
Table E.4: ILM-VP with Hyper-Paramete Tuning. The chosen tuning configurations and the corresponding testing accuracy (%) in ILM-VP. The highest accuracy is marked in bold.
ILM-VP
ILM-VP w/o Tuning
Datasets
AutoVP
w/ Tuning
FullyMap. p 16
mapping number 5, 30
UCF101
Accuracy: 68.4%
Accuracy: 73.1 %
Accuracy: 70.4%
FullyMap. p 16
mapping number = 2, 30
EuroSAT
Accuracy: 96.7%
Accuracy: 96.8%
Accuracy: 96.2%
FreqMap-10 10, p 16
mapping number 5, 30
OxfordllITPet
Aceuracy: 85.1%
Accuracy: 88.2%
Accuracy: 86.7%
27

---

Published as a conference paper at ICLR 2024
E.5
COMPARISON OF AUTOVP AND BLACKVIP
Table E.5: Pu vs. VIackVIP. The testing accuracy (%) of Vu and BlackVIP.
Pets
Cars
Flowers
Food
SUN
DTD
SVHN
EuroSAT
RESISC
CLEVR
UCF
Arg.
AutoVP
88.2
61,8
90.4
82.3
65.4
62.5
92,9
96.8
88.5
82.8
73.1
80.4
Black
89.7
65.6
70.6
86.6
64.7
45.2
44.3
73.1
64.5
36.8
69.1
64.6
Some visual prompting research has delved into black-box setting (Tsai et al. 2020 Oh et al. 2023), where the internal architecture of the pre-trained model remains unattainable during the training process. For instance, BlackVIP (Oh et al. 2023) employs an inpur-dependent prompt designer to generate visual prompts. These prompts are then fed into a black-box model, and gradient approximation strategies are utilized to update the prompt designer. In Table E.5 present a comparison between Auto and Black Since Black uses CLIP as the pre-trained backbone, report results using the same pre- trained model. Vu demonstrates 16% performance increase on average compared to BlackVIP This notable difference might be attributed the variance in update strategies: BlackVIP utilizes SPSA-GC for black-box models, while Au relies on classic gradient descent. However, due to the differing objectives of these two studies, direct comparisons may introduce certain unfairness.
F PERFORMANCE AND RESOURCE UTILIZATION
F.1 COMPARISON OF AUTOVP, LINEAR PROBING, AND FULL TINE-TNNIGG
When comparing the performance of AutoVP to that of linear probing (LP) and full fine-tuning (FF), it's crucial to acknowledge the significant discrepancy in terms of trainable parameter size. As FF involves a parameter size that is roughly 100 to 1000 times larger (as shown in Table F.4). Nevertheless, it's worth noting that the differences in performance between VP and FF are relatively minor in certain datasets. For instance, in the case of EuroSAT, both Vu and FF achieve 96% accuracy (as demonstrated in Table F.). This observation suggests that VP retains its advantages even when faced with such substantial variations in parameter size.
Table F.1: Comparison of Auto Linear Probing (LP), and Full Fine-Tuning (FF). With the EuroSAT dataset and using CLIP as the pre-trained model, the table displays the testing accuracy, execution time, and trainable parameter size associated with each respective method.
Experimental Info.
Au
LP
FF
Accuracy (%)
96.84
94.62
96.78
Execution Time (second)
2448
2370
3081
Trainable Parameter Size (Million)
0.15
0.005
151.28
28

---

Published as a conference paper at ICLR 2024
F.2 COMPUTING RESOURCES
In this section, we provide the execution time (measured on NVIDIA GeForce RTX 3090) and the comparison of trainable parameters of AutoVP. For ease of comparison, use the Flower102 dataset for illustration. In Table F.2 we provide the end-to-end execution time prrramee tuning + 200-epoch training) of Auto VP. When we measure only the 200-epoch training, Au demonstrates its competitiveness in terms of similar or even lower training time (see Table F.3). compared to the state-of-the-art VP baselines (II.M- VP and CLIP-VP The comparison of trainable parameters can also be found in Table
Table F:2: Flowers102 End-to-End Execution Time. For the Flowers 102 dataset, the pre-trained model selected by Au is Swin-T, the output mapping is FullyMap. and the prompt size is 16.
Hyper-Parameter Parameter Tuning 200-Epoch Training
Total Execution Time
Time
146 mins
31 mins
177 mins
Table F.:
Execution Time Comparison. The 200-epoch training time (in minutes) on the Flow- ers 102 dataset varies depending the chosen pre-trained model: Swin-T or CLIP. In both cases, VP utilizes the FullyMap as the output mapping method with the prompt size 16.
Pre-trained model
AutoVP
ILM-VP
CLIP-VP
Linear Probing
Swin-T
31
43
28
CLIP
39
76
38
40
Table F.4: Trainable Parameter Size. The average trainable parameter sizes (million) are calculated across the 12 datasets for different pre-trained models, mapping methods, and baselines.
AutoVP
Linear Probing
Full Finetune
SemanticMap
FreqMap
IterMap
FullyMap
ResNet18
0.15
0.15
0.15
0.20
0.03
11.20
RcsNeXt101-IG
0.15
0.15
0.15
0.20
0.11
86.85
Swin-T
0.15
0.15
0.15
0.20
0.04
27.56
CLIP
0.15
0.15
0.15
0.49
0.03
151.23
29


Reviews:
[{"review_id": "DgPCbhsFjG", "reviewer": "Reviewer_nUAn", "summary": "This paper proposes AutoVP, an end-to-end expandable framework for automating VP design choices along with 12 downstream image classification tasks that can serve as a holistic VP-performance benchmark.", "questions": "I would suggest expanding the testing of visual prompting from image classification to other tasks like detection and segmentation. This would help ensure that the AutoVP framework is versatile and not just fine-tuned for specific tasks. Aim to create a benchmark that evaluates how well visual prompting works generally, across various types of visual tasks.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Clarity and Logic**: The paper is well-structured and presents complex ideas clearly, making it understandable for readers.\n\n**Useful Framework**: AutoVP is introduced as a versatile toolbox that simplifies the development of visual prompts, offering a modular design and comprehensive functionalities.\n\n**Improved Performance**: The models tuned with AutoVP demonstrate a significant performance improvement over previous baselines across various image classification tasks.", "weaknesses": "**Limited Novelty**: The framework largely combines existing methods, which might suggest a wrap-up of previous work rather than introducing new concepts, limiting the perceived novelty of the research.\n\n**Potential Overfitting**: AutoVP uses different settings for different datasets, raising the question of whether these are overfitted to specific tasks and what the implications are for a robust, universal setting.\n\n**Insufficient Analysis of Mapping Methods**: There is a lack of detailed comparison and analysis of the mapping methods used in visual prompting, which is necessary to understand their impact and provide more comprehensive insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5Pif4x4J4X", "reviewer": "Reviewer_PZJW", "summary": "This paper introduces AutoVP, a comprehensive framework for automating VP design decisions. This framework covers several aspects, including optimizing the prompts, selecting suitable pre-trained models, and determining the best output mapping strategies. \nThe AutoVP along with a set of 12 image-classification tasks serves as a benchmark for evaluating VP performance. \n\nAutoVP is shown to outperform existing VP methods and achieves a considerable performance boost when compared to the linear-probing baseline. Above all, AutoVP offers a hyperparameter tuning tool for VP and a benchmark to further its advancements.", "questions": "Please see the weaknesses mentioned above.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper presents its findings with clear figures and detailed statistical reports, making it easier for readers to grasp the results.\n2. This paper does not just present a tool but embarks on a detailed exploration of optimal configurations under various conditions, aiming at proving how different settings affect performance. It also examines the impact of domain similarity on VP performance.", "weaknesses": "1. While VP can potentially be used for a variety of vision tasks, the paper seems to focus primarily on image classification tasks, which may limit its applicability to broader vision problems. Are there any additional results on dense discriminant tasks?\n2. When utilizing CLIP as the pre-trained classifier within the framework, which visual backbone is employed, ViT or ResNet?\n3. About the proposed VP benchmark, why do the authors exclude some widely recognized 2D datasets, such as Caltech101, OxfordPets, StanfordCars, SUN397, EuroSAT, and FGVCAircraft, which are all common-used for 2D image classification task evaluation? What are the criteria for dataset selection?\n4. Table 2 reveals that AutoVP underperforms on **6** datasets out of **12** in the benchmark compared to Linear Probing (LP), e.g., AutoVP is **-6.5%, -10.2%, -12.1%** lower than LP on Flowers102, UCF101, and DTD respectively. The reported average accuracy improvement appears to be significantly influenced by the results from the SVHN dataset, which is **+27.5%**. Could the authors provide additional insights into this discrepancy? Furthermore, similar patterns observed in Tables 6 and 7 suggest that these results may not be consistently solid across varied datasets.\n5. The paper mentions ***std*** for AutoVP, indicating some randomness. It's important to think about how this inconsistency could affect the reliability of AutoVP, especially compared to the more stable method of linear probing. While Table 5 indicates a shorter execution time per run for AutoVP, one might infer that achieving the reported performance could necessitate multiple runs, thereby affecting the efficiency.  This inconsistency and the potential extra time needed make it less practical in certain situations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k28UIxjE0i", "reviewer": "Reviewer_dWYk", "summary": "The paper introduces AutoVP, an end-to-end framework that automatically selects optimal visual prompt (VP) configurations for vision downstream datasets. Tested on 12 datasets, AutoVP demonstrates improvements over the traditional linear probing approach, and on OOD test. The performance enhancements are in line with expectations. The significance of AutoVP lies in its automation of the VP configuration process, offering an extensive study for prompt engineering with gains.", "questions": "No", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. An extensive study for visual prompting on vision model such as ResNeXt, ViT, and CLIP model. \n\n2. AutoVP, by applying a series of established approach, from input scaling, to output label engineering, enables huge gain on the results.", "weaknesses": "1. The paper is evaluated on 12 visual recognition tasks, what about other tasks, given that this is a benchmark paper. Say Object Detection, Depth, Segmetnation. \n\n2. Reviewer appreciate this systematic study in applying all methods of VP and improve results. However, those results are expected.  Learn a bit of new knowledge after reading this, the reviewer would expect in general more surprising finding or impressive knowledge.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ivws073cnU", "reviewer": "Reviewer_aYHC", "summary": "This paper mainly focuses on visual prompting method. The authors propose a new framework which covers a wide range of design space for visual prompting including input scaling, pretrained model, output mapping. The authors introduce several difference techniques for these dimension which are shown to be effective in the experiments.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "1. The proposed method is simple yet effective.\n2. The writing is easy to understand.", "weaknesses": "1. The input scaling technique was discussed in EVP [1]. The authors may have some more discussion about the relationship and difference with this work.\n2. The different choices for pretrained model are almost the same as the original ones in VP. It would be better if the author can mention this in the paper.\n3. I doubt if the proposed FullyMap can be viewed as an output mapping method rather than a variant of linear probing. By using this method, the distribution of trainable parameters is totally different from the original VP-style methods like VP, EVP and BlackVIP [2] whose trainable parameters only concentrate on modifying the input space without the output space. It is for sure that such a method can bring significant improvement in the experiments. Even if you do not use the visual prompt but simply finetune the model with the additional last fc layer, I assume the model can have similar performance to the linear probed one.\n4. I recommend the authors adopt other datasets used in previous VP papers like SUN, CLEVR and RESISC, and compare the proposed method with other baselines like BlackVIP.\n\n\n[1] Wu J, Li X, Wei C, et al. Unleashing the power of visual prompting at the pixel level[J]. arXiv preprint arXiv:2212.10556, 2022.\n[2] Oh C, Hwang H, Lee H, et al. BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023: 24224-24235.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)