Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation
Authors: Shohei Enomoto
Venue: NeurIPS 2025 poster ()
Keywords: visual prompting
adversarial reprogramming
fine tuning
Primary area: deep_learning
TL;DR: Our novel method improves visual prompting accuracy through affine/color transformations and TrivialAugment data augmentation, achieving state-of-the-art results with minimal overhead.

Abstract: Visual prompting (VP) has emerged as a promising parameter-efficient fine-tuning approach for adapting pre-trained vision models to downstream tasks without modifying model parameters. Despite offering advantages like negligible computational overhead and compatibility with black-box models, conventional VP methods typically achieve lower accuracy than other adaptation approaches. Our analysis reveals two critical limitations: the restricted expressivity of simple additive transformation and a tendency toward overfitting when the parameter count increases. To address these challenges, we propose ACAVP (Affine, Color, and Additive Visual Prompting), which enhances VP's expressive power by introducing complementary transformation operations: affine transformation for creating task-specific prompt regions while preserving original image information, and color transformation for emphasizing task-relevant visual features. Additionally, we identify that overfitting is a critical issue in VP training and introduce TrivialAugment as an effective data augmentation, which not only benefits our approach but also significantly improves existing VP methods, with performance gains of up to 12 percentage points on certain datasets. This demonstrates that appropriate data augmentation is universally beneficial for VP training. Extensive experiments across twelve diverse image classification datasets with two different model architectures demonstrate that ACAVP achieves state-of-the-art accuracy among VP methods, surpasses linear probing in average accuracy, and exhibits superior robustness to distribution shifts, all while maintaining minimal computational overhead during inference.
Our code is available at https://github.com/s-enmt/ACAVP.

Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation
Shohei Enomoto
NTT
Tokyo, Japan
shohei wawieeiiennmottttttcom
Abstract
Visual prompting (VP) has emerged as a promising parameter-efficient fine-tuning approach for adapting pre-trained vision models downstream tasks without modi- fying model parameters. Despite offering advantages like negligible computational overhead and compatibility with black-box models, conventional VP methods typi- cally achieve lower accuracy than other adaptation approaches. Our analysis reveals critical limitations: the restricted expressivity of simple additive transforma- tion and tendency toward overfitting when the parameter count increases. To address these challenges, we propose ACAVP (Affine, Color, and Additive Visual Prompting), which enhances VP's expressive power by introducing complementary transformation operations: affine transformation for creating task-specific prompt regions while preserving original image information, and color transformation for emphasizing task-relevant visual features. Additionally, we identify that overfitting is a critical issue in VP training and introduce TrivialAugmen as an effective data augmentation, which not only benefits our approach but also significantly improves existing VP methods, with performance gains of up to 12 percentage points on certain datasets. This demonstrates that appropriate data augmentation is universally beneficial for VP training. Extensive experiments across twelve diverse image classification datasets with two different model architectures demonstrate that ACAVP achieves state-of-the-e art accuracy among VP methods, surpasses linear probing in average accuracy, and exhibits superior robustness to distribution shifts, all while maintaining minimal computational overhead during inference. Our code is available at hhttts://// htt///gttttubb/ com/s s- enmt/ ACAVP
Introduction
Visual prompting (VP) [Bahng et al. 2022, Elsayed et al., 2019) has emerged as a promising parameter-efficient fine-tuning technique for adapting pre-trained vision models downstream tasks. By learning task-specific noise applied to input images while keeping the model parameters frozen, VP offers significant advantages: it introduces negligible computational overhead and can be applied to black-box models where internal parameters are inaccessible Tsai et al. (2020). Oh et al. (2023).
Despite these advantages, VP methods typically achieve lower accuracy than full fine-tuning BBhnng et al. 2022]. A straightforward approach to improving VP accuracy is to increase the size of the prompt region, thereby enhancing the expressive power through increased parameterization. Theoretically, since the transformation space with increased parameters becomes a superset of the original transformation space, the approximation error should decrease with more parameters [Cai et al. 2024b). However, as our preliminary experiments show, this approach is ineffective. As illustrated in Table increasing the number of VP parameters does not improve training accuracy,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and more importantly leads to lower test accuracy, indicating overfitting issues rather than enhanced expressive power.
Our analysis reveals two critical limitations in VP: (1) simply adding more parameters with the ditional additive transformation has inherent lim- itations in its expressive power, and (2) increasing parameter count leads to overfitting issues. This leads us key question. How can we enhance the expressive power of VP while effectively pre- venting overfitting?
Table I: Results of preliminary experiments com- paring performance between padding-type noise addition and image-sized noise addition.
Image
To address this challenge, we propose ACAVP (Affine, Color, and Additive Visual Prompting), novel VP transformation design that enhances ac- curacy while maintaining minimal computational overhead. ACAVP incorporates two parameter- ized image transformation techniques into exist- ing VP methods: affine and color transformations. The affine transformation component generates VP regions through operations such as resizing and rotation while preserving the original image information. In contrast, the color transformation component enhances the original image information through adjustments brightness and contrast. Importantly, both components retain the advantages of VP technology because of their negligible computational cost compared to model inference. Furthermore, to address the overfitting problem, we comprehensively evaluated various regularization techniques for VP training. After experimenting with several strategies, including dropout, MSE loss regularization, and weight decay, we found that data augmentation methods, particularly TrivialAugment [Müller and Hutter] 2021], are most effective for mitigating overfitting in VP training. TrivialAugment provides a low-cost yet powerful approach to increase input diversity during training, significantly stabilizing the learning process and improving generalization performance.
Params
23,280
50,176
Train Acc
97.53 0.34
97.35 +: 0.18
Test Acc
93.65 + 0:44
89.52 +: 0.12
We evaluated ACAVP's effectiveness across twelve image classification datasets using two different architectures. Our method outperforms existing VP approaches and achieves higher average accuracy than linear probing. Additionally, ACAVP demonstrates superior out-of-distribution (OOD) perfor- mance due to improved generalization capabilities. The computational overhead of ACAVP negligible compared to the inference time of the recognition model.
Our main contributions are follows:
We propose ACAVP, novel VP method that enhances the expressive power of visual prompting by introducing complementary transformation operations: affine transformation for creating task- specific prompt regions while preserving original image information, and color transformation for emphasizing task-relevant visual features. This approach achieves state-of-the-arr accuracy among VP methods while maintaining minimal computational overhead.
We identify and address the critical overfitting problem in VP by empirically evaluating regulariza- tion techniques. We demonstrate that TrivialAugment effectively mitigates overfitting in ACAVP and existing VP methods, providing a general technique for improving VP performance.
We evaluated our approach through extensive experiments across twelve diverse image classification datasets and two different model architectures. Our experiments demonstrate that ACAVP achieves the highest average accuracy among VP methods, surpasses linear probing in average accuracy, and exhibits superior robustness distribution shifts in out-of-distribution scenarios.
Related Work
Visual prompting (VP) [Bahng et al., 2022 has emerged as a promising parfmeter-efficient fine- tuning paradigm. First introduced as adversarial reprogramming (AR) by Elsayed et al. [2019), VP adapts pre-trained models to downstream tasks by applying learnable prompts to input images while keeping the model parameters frozen. This approach enables efficient adaptation without modifying the underlying model architecture.

---

Transformation design for VP. Although the original VP adds padding-type type noise to the image, several studies have focused on improving its transformation design. WWu al. [2024] introduced enhanced visual prompting (EVP), which preserves original image information by resizing the image and adding prompts around it. Building on this concept, [sao et al. [2024) proposed AutoVP, an end-to-end framework that automates VP design choices and introduces learnable input scaling parameters to optimize the resize intensity dynamically. In a different direction, Jin et al. (2025) de- veloped wa-Rank matrix multiplication for visual prompting (LoR-VP), which uses low-rank matrix decomposition to ensure that visual prompts influence all image patches while considering induced biases between patches. Rather than using single prompt for entire task, several approaches generate customized prompts for individual samples. Oh et al. 2023) proposed Coordinator, which uses recoder-decoder network generate sample-specifie prompts. Similarly, Cai et al. [2024b] proposed sample-specific multi-channel masks (SMM) that generate appropriate prompt regions for each individual sample. [ho et al. 2024] expanded this concept by generating prompts in both spatial and frequency domains using a network for sample-specific adaptation. While sample-specific VP approaches achieve high expressive power, they significantly increase inference-time computational overhead, undermining one of VP's primary advantages. In contrast, our work focuses on task-level visual prompting while addressing its limitations.
Learning strategies for VP. Researchers have also explored improved learning strategies for VP.
Wu et al. [2024) incorporated techniques from adversarial attacks, specifically input diversity (Xie
et al. 2019 and gradient normalization [Goodfellow et al., 2013]. to improve training stability. Cai
al. 2025 proposed attribute-based visual reprogramming (AttrVR), leveraging descriptive and
distinctive attributes to mitigate ambiguities caused by fixed template prompts and promote more context-aware alignment between images and attributes. Zhang al. (2024) introduced transferable visual prompting (TVP), which learns a single VP to improve performance across diverse multimodal large language models. the non-CLIP model, random class mapping is used to align the pre- trained model classes with the downstream tasks' classes. WW et al. [0024] proposed a method that first infers the pre-trained model on downstream dataset and then performs label mapping according to its output distribution. Chen al. [2023] introduced iterative label mapping (ILM), which automatically remaps source labels to target labels, progressively improving VP performance on target tasks. Taking probabilistic approach, Cai et al. [2024a] proposed bayesian-guided label mapping (BLM), which constructs a sequentially updated stochastic label mapping matrix quantifying pairwise relationships between pre-traine pre-trained labels and downstream labels. In this study, show that VP training is less accurate due to overfitting, and that countermeasures, especially data augmentation, the most effective learning strategy.
We propose a novel VP design that enhances expressive power while minimizing the computational cost increase in inference time. Additionally, we introduce data augmentation techniques to mitigate overfitting, a common issue in VPs. Our approach balances expressive power and computational efficiency, maintaining VP's key advantages while improving performance.
3 Proposed Method: ACAVP
We propose a novel VP approach, ACAVP (Affine, Color, and Additive Visual Prompting), to enhance VP by introducing additional transformation operations that address its limitations. Our investigation begins with identifying the primary constraints of existing VP techniques and then proposing solutions that maintain the low computational overhead characteristic of VP methods. Figure
[ shows an overview of ACAVP. The theoretical justification for our approach is provided in Appendix B. where we formally analyze the hypothesis spaces of existing VP methods and ACAVP, demonstrating that ACAVP has a smaller approximation error, which explains its superior performance. The implementation details are provided in Appendix C
3.1 Preliminaries
VP is a parameter-efficient fine-tuning technique that adapts pre-trained recognition models to downstream tasks by applying learnable prompts to input images while keeping the model parameters frozen [Bahng et al. 2022]. During training, VP learns a prompt that minimizes the task-specific loss of the recognition model. At inference time, the learned prompt is applied to input images before model prediction. Formally, given an input image x E
traditional VP transforms
3

---

f
L(P,y)
Affine
Forward
A ffine Prompt
Color Prompt
Additive Prompt
a
b
Backward
0
Figure 1: Overview of ACAVP. The input image undergoes three transformations: first, affine transformation using the affine prompt, followed by color transformation using the color prompt, and finally an additive transformation using the additive prompt. The resulting VP image is fed into frozen recognition model, and the loss backpropagated to update all three prompt parameters.
it into prompted image 3 E R3xHxW using the following equation: 3 = x + M @ 8, where M E {0, 1) is a binary mask indicating the prompt region, E |2xHXx is the learnable prompt parameter, and denotes element-wise multiplication.
VP has advantages such as almost zero computational overhead during inference and applicability to black-box recognition models (including cloud APIs) [Oh et al. 2023 Tsai et al. 2020 in order to maintain the original model architecture and parameters. However, as our preliminary experiments show, VP suffers from two significant limitations: limited expressive power due to its reliance simple additive transformations and a tendency to overfit when the parameter count increases.
3.2 Problem Analysis
Our preliminary experiments revealed two significant limitations of conventional VP approaches. First, increasing the number of VP parameters by expanding the prompt size theoretically reduces the approximation error, but does not improve the test training accuracy. This suggests that the additive transformation alone has inherent limitations in its expressive power. Second, as the number of parameters increases, the VP model tends overfit, reducing test accuracy. Based these observations, we hypothesized that: (I) introducing different types of transformations beyond addition could enhance the expressive power of VP, and (2) introducing effective regularization strategies could mitigate overfitting issues.
3.3 ACAVP Design
To address these limitations, we propose ACAVP (Affine, Color, and Additive Visual Prompting). an enhanced VP method incorporating affine transformation and color transformation into the VP design, accompanied by effective regularization through data augmentation techniques. Our proposed method transforms the input image according to:
T = Affine A)
where Affine(,, A) represents the affine transformation with learnable affine prompt E R3333 R3xHxW is the learnable color prompt, and 8 E R3*H*W is the learnable additive prompt. Unlike conventional VP methods, M E {o, 1 is dynamically generated after the affine transformation is applied, where M has value for pixels where all three channels are zero, and 0 elsewhere. This ensures the additive prompt is only applied to the empty regions that the affine transformation creates. This formulation extends the transformation space beyond simple addition, enabling more expressive prompts while maintaining the computational efficiency of VP.
The affine transformation prompt generates task-specific feature regions while preserving the original image information. The affine transformation enables geometric modifications, including rotation, translation, shearing, and scaling, providing richer set of transformations than simple addition. For an input point (u, v), the affine transformation with transformation matrix A computes its new coordinates (ù, 1) follows:

---

& cos 0 + sh, sin 0 Sy sin i + shy cos
-8 sin 0 + shz cos Gr tg
by cos 1 + shy sin O
(2)
To ensure stable training and prevent extreme transformations that might destroy the original image information, we constrain the transformation parameters using activation functions: = tanh(t) RR, i = (0) Rg. 5 sigmoid(s), and sh tanh(sh) Ras- Here, f represents learnable a learnable rotation parameter, s refers to learnable scaling parameters, and sh denotes learnable shearing parameters. The hyperparameters R,, Rø, and Rsh control the maximum range each transformation. The default values for these hyperparameters are set to R₂ 0.05, Rg Righ 0.1.
translation parameters,
The color transformation prompt further enhances the expressive power of our VP by adjusting image properties such as brightness and contrast. This is achieved through learnable parameter R applied multiplicative factor to the transformed image. The color transformation allows the prompt to emphasize or attenuate specific features in the original image, potentially highlighting task-relevant information.
To prevent excessive color transformation that might corrupt the original image information, we constrain the values using: ð (o) Rg: where Rg is a hyperparameter controlling the maximum magnitude of color transformation, with a default value of Rs = 6.0. This ensures that the color transformations remain within reasonable range while still providing sufficient flexibility to enhance task-relevant features.
3.4 Overfitting Mitigation
As shown our preliminary experiments, increasing the number of VP parameters theoretically reduces the approximation error, but causes overfitting. This observation led us to hypothesize that effective regularization strategies would be crucial for our enhanced ACAVP approach with its additional transformation parameters. To identify the most effectivee overfitting mitigation technique, we comprehensively evaluated various regularization approaches applied to VP training. Specifically, investigated four categories of techniques: data augmentation, dropout applied after feature extraction layer, MSE loss between original and prompted images as an additional regularization loss term, and weight decay. Our experimental results, detailed in Section 4.4 revealed that data augmentation methods consistently provided the most substantial improvements in generalization performance. Among the various augmentation techniques evaluated, particularly effective for VP training. TrivialAugment applies a randomly selected augmentation operation with random magnitude to each input image, providing sufficient diversity mitigate overfitting while maintaining implementation simplicity. While previous VP approaches have employed basic augmentations like random crop and horizontal flip, found that more sophisticated augmentation strategies could significantly improve the performance of VP.
emerged as
4
Experiments
To evaluate the effectiveness ACAVP, we conducted extensive experiments across 12 diverse image classification datasets using two different model architectures. We compared ACAVP against zero- shot inference (ZS) with CLIP [Radford et al. 2021] and state-of-the-art VP methods (VP [Bahng al. 2022 EVP [Wu al. 2024 and Auto VP [Tsa al. 20244) ). All experiments were repeated three times, and we report the average accuracy and standard error. Each dataset was split into training, validation, and test sets, and we used the checkpoint with the highest accuracy on the
validation set for final testing. Detailed experimental settings are provided in Appendix D
4.1 Main Results
We evaluated ACAVP using a ViT-B32-based CLIP model [Radford al., 2021 Dosovitskiy et al., 2021 across 12 diverse image classification datasets. Table shows the comparative performance results. ACAVP demonstrates superior accuracy on most datasets compared existing VP methods, achieving average accuracy of 83.18% which significantly outperforms VP, EVP, and Auto The
5

---

Table 2: Classification accuracy comparison on image classification datasets using CLIP V/T-B/32.
Bold and underlined values denote the first and second highest accuracy, respectively. The LP and
FT results refer to Bahng et al. [022], Tsao et al. 2024], Tang et al. [2024).
Method
ZS
VP
EVP
Au
ACAVP
LP
FT
CIFARIO
88.93
93.65
uoe
95.87 0.6
95.39
0.0
96.27
95.00
95.80
CIFAR100
63.09
74.71
78.33
0.
79.89
80.06
80.00
82.10
CLEVR
20.70
76.40
+ 0.56
76.82
0.2
79.19
0.22
77.19
0.17
66.00
94,40
DTD
42.91
57.19
0.34
60.89
0.57
62.81
0.12
67.43
0.83
74.60
72.30
EuroSAT
39.02
96.36
ner
97.27
2 0.08
97.32
2.10
97.51
0.00
95.30
97.90
Flowers
61.63
67.83
77.76
82,69
89.36
96.90
97.40
Fond
79.67
78.92
77.48
11000
76.53
2.00
77.86
one
84.60
80.50
GTSRB
18,18
91.38
0.28
76.17
2.40
92.50
0.14
92.04
0.35
85,80
98,90
Pets
85,96
86.51
+ 0.30
87.85
0.06
86.71
0.26
87.55
0.22
89.20
88.50
SUN
60.55
60.39
0.12
63.51
0.00
61,66
0.10
64.95
75,00
64.00
SVHN
15.30
91.61
86.69
92.94
91.81
65.40
95.70
UCF
61.46
65.68 6.28
67.69
0.50
70.88 =: 0.11
76.11 0.34
83.30
80.90
Average
53.12
78.39
78.86
81.54
83.18
82.59
87.37
consistent improvements across diverse datasets highlight the enhanced generalization capability of ACAVP. Notably, ACAVP shows remarkable improvement on the Flowers dataset, with absolute gains of 6.67 percentage points over Au and 21.53 percentage points over the VP. This substantial improvement suggests that our affine transformation and color transformation components are particularly effective for datasets with fine-grained visual features, such as the intricate textures and shapes found in flower images. The ability to capture such nuanced visual characteristics demonstrates the enhanced expressive power of ACAVP. Interestingly, ACAVP surpasses the average accuracy of Linear Probing (LP), which modifies the model parameters. VP methods, including ours, demonstrate particularly strong performance on datasets that differ significantly from the model's pretraining distribution. For instance, on specialized datasets such as GTSRB (traffic signs) and SVHN (street view house numbers), VP approaches substantially outperform LP. This pattern aligns with previous findings that VP is especially effective for adapting domains that deviate from the source distribution used during pretraining [Bahng et al. 2022]. In summary, our experimental results demonstrate that incorporating affine transformation and color transformation alongside overfitting mitigation techniques effectively addresses the limitations of conventional VP approaches. These results validate our hypothesis that diversifying the transformation space and mitigating overfitting are crucial factors for enhancing the effecttieness of VP techniques.
We further evaluated ACAVP using ResNet50 [He et al., 2016) pre- pre-trained the ImageNet-II Ik dataset (Russakovsky et al. 2015) across the same 12 image classification datasets. This evaluation aims to demonstrate the veneralizabiligg of our approach across different model architectures and pretraining strategies. Table 3 shows the results. Similar to our findings with the CLIP model, ACAVP outperforms baseline approaches on many datasets and achieves the highest average accuracy among all VP methods. Specifically, without label mapping optimization (left columns), ACAVP achieves the highest average accuracy, significantly outperforming the second-best method by 15.01 percentage points and 21 1,500 percentage points on the Flowers and GTSRB datasets, respectively. Similarly, with label mapping (denoted by in the right columns), ACAVP achieves the highest average accuracy. While all methods benefit from label mapping, our approach maintains its superior performance, suggesting that the benefits of our enhanced transformation space are complementary to those provided by label mapping. The consistent relative performance across both settings indicates that ACAVP's improvements stem from fundamental enhancements the VP framework rather than interactions with specific implementation details. These results demonstrate that ACAVP generalizes effectively across different model architectures and pretraining strategies. The consistent performance improvements observed with both CLIP and ResNet50 backbones suggest that the limitations of conventional VP approaches are fundamental rather than model-specific and that our enhancements to the transformation space and learning methodology principledly address these limitations.

---

Table 3: Classification aceuracy comparison on 12 image classification datasets using ResNet50 pre- rre-trained on ImageNet-Ie lk. indicates results with label mapping proposed by Wu et al. [2024].
Method
LVP
AutoVP
ACAVP
VP
EVP
Vp
ACAVP
CIFARIO
55.09 n
54.28 2.00
45.52 aM
53.48 nm
64.09 2.
67.93 2a
62.77 2.s
72.92 a0
CIFAR100
10,95 1
7.40 122
6.80 136
14.84 tue
27.69 an
32.30 025
29.44 am
33.71 a5
CLEVR
42.19 1
40.62 10.7
28.90 1.e
32.23
40.23 10
37.88 03
29.16 a.0
38,95 2.
DTD
13.46
15.98
12.59
9.34
30.65 2015
35.62
30.04 1:
34.16
EuruSAT
79.06 uee
78.44 o0
68.79
76.70
82.18 2:30
79.94
71.82 2:9
83.04
Flowers
8.40 9:03
10.37 0.0
10.22
25.38 toa
12.13 0.0.
16.59 m:
14.60 0:3.4
19.77 =0
Food
4.22 0.17
3.24 -.0
2.82 334
6.69 + c.e
7.66 um0
7.14 +..
7.18 w
8,55 + 020
GTSRB
38.87
30.26
28.17
60.37
45.89 0:0
34,93 039
37.66
45.77 088
Pets
5.76
7.47
6,92
12.23
64.80
68.66
66.68 a
68.23
SUN
0.87 oss
1.10 cos
0.60
1,50 +:000
7,93 +as
8.73 +
7.74 +
7.33
SVHN
66.34 ozz
48.21 020
46.60 1:5
62.21 0ua
67.84 0:021
49.17 a:4
45.33 1:44
68.82 0.444
UCF
8.19 1:033
7.28 2.0
6.53 3:10
13.11 1.0
24.50 1:021
27.56 o0r
26.14 1:0
28.14 0.5
Average
27.80
25.39
22.04
30.67
39.63
38.87
35.71
42.45
Table 5: An ablation study shows the contribution of different transformation components. Affine refers to affine transformation, Additive to additive transforma- tion, and Color to color transformation. All methods use TrimialAugment for fair comparison.
Table 4: Classification accuracy on the corrupted datasets using CLIP ViT-B/32. Classification accuracy means the aver- age accuracy at 15 different types and 5 levels of severity.
Transformation
CIFARIO
Flowers
Method
CIFARIO-C
CIFAR 100-C
88,93
61,63
Affine
90.17
57.42 +
ZS
70.9
42.59
Color
92. 14 1
62.92 1
VP
76.65 0.4
50.52 00012
Affine + Color
92.46 2:0
60.58 1 aos
EVP
80.97
0.14
55.17
Resize + Additive
96.00 1 0.05
79.24 1 a:x
AutoVP
79.72 0.2
56.34 +0.15
Resize Additive Color
96.05
80.74
ACAVP
83.98
0.30
58.68 20
Affine + Additive
96.31
88.70 +
Affine + Color + Additive
96.27 + 0.00
89.36 + 0.9
4.2 Robustness to Distribution Shift
We evaluated the robustness of ACAVP to distribution shifts by comparing its performance against baseline approaches on artificially corrupted datasets. Distribution robustness critical aspect of adaptation methods in real-world scenarios where test data often differs from training data due to various forms of noise and corruption. For this evaluation, we tested various VP approaches on the CIFARIO-C and CIFAR 100-C datasets, which apply 15 common types of image corruptions (including noise, blur, weather, and digital distortions) at different severity levels to the original CIFAR10 and CIFAR100 test sets, respectively (Hendrycks and Dietterich] 2019]. These standardized cоruptiion benchmarks provide a systematic way to assess model robustness under various distribution shifts. The results are shown in Table ACAVP consistently outperforms all baseline approaches by significant margins on both corrupted datasets. Notably, the performance improvement of ACAVP is more pronounced Table 2 For example, on the CIFAR 10-C dataset, ACAVP achieves a 7.33 percentage points improvement VP, which is higher than the 2.62 percentage points improvement observed on the original CIFAR10 10 dataset. This enhanced robustness to distribution shifts suggests that the diverse transformation space introduced by ACAVP, combined with overfitting mitigation techniques, enables learning more generalizable transformations. These results highlight an important practical advantage of ACAVP: improved robustness to real-world distribution shifts, which is often crucial for the deployment of adaptation methods in dynamic and unpredictable environments.
out-of-distribution data compared in-distribution performance (refer to
4.3 Ablation Study
We conducted an ablation study to analyze the contribution of individual components in ACAVP on the CIFARIO 10 and Flowers datasets. This analysis helps identify which transformations are most effective and how they interact with each other. Using ZS (no transformation) and EVP (which
7

---

Table 6: Comparison of overfitting mitigation techniques on image-sizes VP with CLIP ViT-B/32.
Values in parentheses show accuracy gains from using overfitting mitigation techniques.
Method
CIFARIO
Flowers
Dropout
88,50 # oss (-1.02)
53.57 = =0 (-0.28)
MSE Loss
90.26 + om (+0.74)
53.74 =0 (-0.11)
Weight decay
91.25 +:00 (+1.73)
55.37 =o (+1.52)
TrivialAugment
91.56 1:m (42.04)
70.11 (+16.26)
IPMix
90.85 1om (+1.33)
66.03 1 (+12.18)
RandAugment
91.52 1000 (+2.00)
72.88 = mm (+19.03)
applies an additive transformation after resizing) as the baselines, we evaluated six variants: affine transformation alone (Affine), color transformation alone (Color), affine and color transformations (Affine + Color), EVP with color transformation (Resize + Additive + Color), EVP with affine transformation replacing the resize operation (Affine + Additive), and ACAVP that combines affine transformation, color transformation, and additive transformation (Affine + Color + Additive). To ensure a fair comparison, we incorporated TrimialAugment including the EVP baseline. The results are shown in Table 5 Individual transformations show varying effectiveness across datasets: the affine transformation improves performance on the CIFARIO 10 dataset but shows reduced performance transformation consistently outperforms ZS accuracy on both datasets. Combining affine and color transformations yields further improvements. On the Flowers dataset, which contains rich textural and color information, the performance gains are particularly illuminating. The variant with color transformation (Resize + Additive Color) outperforms EVP by 1.50 percentage points, while the variant with affine transformation (Affine + Additive) exceeds EVP by 9.46 percentage points. These results demonstrate the significant contributions of both color transformation and affine transformation to accuracy, with affine transformation having a more substantial impact on the Flowers dataset. Notably, our full method, which incorporates both affine and color transformations, achieves the highest accuracy, surpassing by 10. percentage points on the Flowers dataset. This suggests that these transformations are complementary rather than competing, and their combination leads to more effective adaptation. The affine transformation likely helps in focusing on the most discriminative regions of flowers, while the color transformation enhances the representation of subtle color variations that are critical for flower classification. On the CIFAR10 10 dataset, all variants show improvements over ZS, though the differences are less pronounced than on the Flowers dataset. This smaller gap is expected since the CIFAR10 features less complex visual patterns and color variations than the Flowers. Nevertheless, the improvements across both datasets confirm that each component contributes positively to the overall performance of our method. This ablation study validates our design choices and demonstrates that both the affine and color transformation components contribute meaningfully to improving VP performance.
the training process for all methods,
the Flowers dataset compared to ZS, while the color
4.4 Effectiveness of Overfitting Mitigation Strategies
We demonstrate experimentally that mitigating overfitting plays a crucial role in VP training. Our preliminary experiments showed that VP approaches, particularly those using image-sized noise additions, exhibit strong overfitting tendencies that limit performance. Here, we investigate various strategies to address this limitation. We first experimented applying various overfitting mitigation techniques to the image-sized VP approach, which showed overfitting tendencies our preliminary studies. The techniques we evaluated include: dropout layer after the feature extraction layer, MSE loss minimization between the original and prompted images, weight decay for parameter regularization, and three data augmentation methods, TrinialAugment,, RandAugment [Cubuk et al. 2020). and IPMix [Hang et al. 2023], which increase input diversity. The results are shown in Table 6 Dropout and MSE show no significant improvement over the baseline. Weight decay demonstrates a modest accuracy improvement. Most notably, data augmentation techniques yield substantial performance gains, with TrivialAugment and RandAugment showing powerful improvements on both datasets. These results strongly suggest that overfitting mitigation is critical for VP training, with data augmentations being especially effective.
Next, we investigated whether Trivial Augment, the effective and computationally efficient regulariza- tion technique from our first experiment, could improve the performance of existing VP basclines.
8

---

Table 7: Effect of TugmeelAugment on existing VP methods with CLIP ViT-B/32. Values in parentheses show accuracy gains from adding Tugnia.ugment.
Method
CIFARIO
Flowers
VP
94.97 # 200 (+1.32)
79.84 84 = (+12.01)
EVP
96.00 + oos (+0.13)
79.24 = a18 (+1.48)
Au
96.16 +00 (+0.77)
85.83 0.58 (+3.14)
The results, shown in Table demonstrate consistent accuracy improvements across all VP baselines when TrivialAugment is applied. The improvements particularly pronounced for the standard VP approach on the Flowers dataset, with a substantial 12.01 percentage point accuracy gain. Interest- ingly, with TuivialAugment, the standard VP achieves accuracy comparable EVP, despite EVP's additional resize operation. This suggests that the resize operation in EVP may have been functioning primarily implicit regularizer by preserving more of the original image information and limiting overfitting. However, this advantage diminishes with explicit input diversity through TrivialAugment, indicating that VP and EVP may have similar expressive power when properly regularized. These findings strongly support our approach of diversifying the transformation space while addressing overfitting. They suggest that the performance limitations of conventional VP methods stem not only from restricted transformation operations but also from their susceptibility to overfitting, By combining more expressive transformation space with effective regularization techniques, ACAVP addresses both limitations simultaneously, leading to substantial performance improvements across diverse datasets.
4.5 Computational Efficiency Analysis
A key advantage of VP is their minimal computa- tional overhead during inference, they modify only the input images without altering the recog- nition model. While ACAVP enhances accuracy by incorporating affine and color transformations, these additional operations naturally introduce some computational costs. In this section, we measured the inference time of different VP meth- ods. For comparison purposes, we also measured the inference time of the recognition model it- self (CLIP ViT-B/32) and a neural network-based VP approach called Coordinator [Oh et al. 2023) which uses an encoder-decoder architecture to gen- erate visual prompts. The results are presented in As expected, ACAVP requires more computation time than VP and EVP. However, when compared the model's inference time, ACAVP adds only 6% of the model's computational cost, which is negligible in practical applications. Notably, the Coordinator has a longer inference time than the recognition model inference. This substantial overhead undermines one of the key benefits of VP approaches: their minimal impact on inference efficiency. Our approach strikes an optimal balance be- tween computational efficiency and transformation expressivity. Using linear transformations (affine and color) rather than complex neural networks, we achieve significant improvements in accuracy with only modest increase in computational cost. This makes ACAVP particularly attractive for real-world applications where performance and efficiency are important considerations. These results highlight an important design principle for VP approaches: while enhancing the transformation space is crucial for improving accuracy, the choice of transformation operations should consider their computational implications. Linear transformations like those used ACAVP offer a favorable trade-off: They provide substantial expressive power with minimal computational overhead.
Table 8: Inference time comparison across dif- ferent methods using H100 GPU with batch size 500. Relative Time is normalized the CLIP ViT-B/32's inference time.
Method
Time (s)
Relative Time
VP
2.88 x 10
0.00
EVP
1.38 x 10-4
0.00
Au
1,77 x 10-2
0.60
ACAVP
1.63 10-3
0.06
Coordinator
2.01 x 10-1
6.85
Model Inference
2.93 x 10-2
1.00
Table 8
Conclusion
In this paper, we addressed the limitations of conventional visual prompting (VP) techniques by introducing ACAVP, novel framework that enhances VP's expressive power and mitigates over-

---

fitting issues. Our approach builds upon the existing VP paradigm by incorporating two additional transformation operations: affine transformation, which creates task-specific prompt regions while preserving original image information, and color transformation, which emphasizes task-relevant features through brightness and contrast adjustments. Furthermore, we demonstrated that data aug- mentation techniques, particularly TrivialAugment significantly improve generalization performance for VP methods by mitigating overfitting tendencies. Our extensive experimental evaluation across 12 diverse image classification datasets using both CLIP ViT and ResNet50 architectures validates the effectiveness of ACAVP. ACAVP consistently outperforms existing VP methods and achieves higher average accuracy than linear probing, which modifies model parameters. Notably, method exhibits superior robustness to distribution shifts, as demonstrated by its performance on corrupted datasets. These improvements are achieved while maintaining VP's key advantage of minimal computational overhead during inference, with ACAVP adding only 6% to the model's inference time. The success of ACAVP highlights the importance of transformation expressivity and regularization in VP training.
References
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint arXiv:2203. 17274, 2022.
Lukas Bossard, Matthieu Guillaumin, and Luc Van Gool. Food-101-mintt discriminative compo- nents with random forests. In European Conference on Computer Vision (ECCV), 2014.
Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, and Feng Liu. Bayesian-guided label mapping for visual reprogramming. In Advances in Neural Information Processing Systems (NeurIPS), 2024a.
Chengyi Cai, Zesheng Ye, Lei Feng. Jianzhong Qi, and Feng Liu. Sample-spent masks for visual seprogramming-haser prompting. In International Conference on Machine Learning (ICML), 2024b.
Chengyi Cai, Zesheng Ye, Lei Feng, Jianzhong Qi, and Feng Liu. Attribute- based visual reprogram- ming for image classification with clip. In International Conference on Learning Representutions (ICLR), 2025.
Aochuan Chen, Yuguang Yao, Pin-Ye Yu Chen, Yihua Zhang, and Sijia Liu. Understanding and improving visual prompting: A label-mapping perspective. In /EEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2023.
Blang-CChehh Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint arXiv: 2017.
Wonwoo Cho, Kangyeol Kim, Saemee Choi, and Jaegul Choo. Training spatial-frequent visual prompts and probabilistic clusters for accurate black-box transfer learning. In ACM International Conference on Multimedia (ACMMM), 2024.
Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describ- ing textures in the wild. In IEEE/CVE Conference on Computer Vision and Puttern Recognition (CVPR), 2014.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, 2020.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general- pprppsse vision-language models with instruction tuning. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
Gamaleldin F Elsayed, Ian Goodfellow, and Jascha Sohl-Dickstein. Adversarial reprogramming of neural networks. In International Conference Learning Representations (ICLR), 2019.
10

---

Mark Everingham, SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 2015.
Yulu Gan, Xianzheng Ma, Yihang Lou, Yan Bai, Renrui Zhang, Nian Shi, and Lin Luo. Decorate the newcomers: Visual domain prompt for continual test time adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). 2023.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. In International Conference on Learning Representations (ICLR), 2015.
Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang. Siyuan Qi, and Dongfang Liu. R 2 vpt: An effective and efficient approach for visual prompt tuning. In IEEF/CVF International Conference on Computer Vision (ICCV), 2023.
Cheng Han, Qifan Wang. Yiming Cui, Wenguan Wang. Lifu Huang, Siyuan Qi, and Dongfang Liu. Facing the elephant in the room: Visual prompt tuning or full finetuning? In International Conference on Learning Representations (ICLR), 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019.
Sebastian Houben, Johannes Stallkamp, Jan Salmen, Mare Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The German Traffic Sign Detection Benchmark. In International Joint Conference on Neural Networks (IJCNN), 2013.
Zhenglin Huang, Xiaoan Bao, Na Zhang, Qingqi Zhang, Xiao Tu, Biao Wu, and Xi Yang. Ipmix: Label-preserving data augmentation method for training robust classifiers. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference Computer Vision (ECCV), 2022.
Can Jin, Ying Li, Mingyu Zhao, Shiyu Zhao, Zhenting Wang, Xiaoxiao He, Ligong Han, Tong Che, and Dimitris N. Metaxas. Lor-VP Low-rank visual prompting for efficient vision model adaptation. In International Conference on Learning Representations (ICLR), 2025.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Michael J Kearns and Umesh Vazirani, An introduction to computational learning theory. MIT press, 1994.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Master's thesis, University of Tront, 2009.
Dongxu Li, Junnan Li, and Steven Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yarn Fu, and Alexander C Berg. Ssd: Single shot multibox detector. In European Conference Computer Vision (ECCV), 2016.
Samuel G Müller and Frank Hutter. Trivialaugment: Tuning-free yet state-of-the-ar data augmenta- tion. In JEEE/CVF International Conference on Computer Vision (ICCV), 2021.
11

---

Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning, 2011.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics & Image Processing, 2008
Changdae Oh, Hyeji Hwang, Hou-young Lee. YongTaek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, and Kyungwoo Song. Blackvip: Black-box visual prompting for robust transfer learning. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnacve. Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2012.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision (IJCV), 2015.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: visual explanations from deep networks via gradient-based localiza- tion. International journal of computer vision (IJCV), 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprini arXiv: 1009: 1556, 2014.
Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: 121.0402 2012.
Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. FusionBench: A Comprehensive Benchmark of Deep Model Fusion, 2024.
Yun-Yun Tsai, Pin- Yu Chen, and Tsung-Y Yi Ho. Transfer learning without knowing: Reprogramming black-box machine learning models with scarce data and limited resources. In International Conference Machine Learning (ICML), 2020.
Yun-Yun Tsai, Chengzhi Mao, and Junfeng Yang. Convolutional visual prompt for robust visual perception. Advances in Neural Information Processing Systems (NeurlPS), 2023.
Hsi-Ai Tsao, Lei Hsiung, Pin- Yu Chen, Sijia Liu, and Tsung- Yi Ho. AutoVP: An Automated Visual Prompting Framework and Benchmark. In International Conference on Learning Representations (ICLR), 2024.
Junyang Wu, Xianhang Li, Chen Wei, Huiyu Wang, Alan Yuille, Yuyin Zhou, and Cihang Xie. Unleashing the power of visual prompting at the pixel level. Transactions on Machine Learning Research (TMLR), 2024.
Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2010.
Cihang Xie, Zhishuai Zhang, Yuyin Zhou, Song Bai, Jianyu Wang, Zhou Ren, and Alan I. Yuille. Improving transferability of adversarial examples with input diversity. In IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
12

---

Runjia Zeng, Cheng Han, Qifan Wang, Chunshu Wu, Tong Geng, Lifu Huangg. Ying Nian Wu, and Dongfang Liu. Visual fourier prompt tuning, Advances in Neural Information Processing Systems (NeurIPS), 2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training In IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Yichi Zhang, Yinpeng Dong, Siyuan Zhang, Tianzan Min, Hang Su, and Jun Zhu. Exploring the transferability of visual prompting for multimodal large language models. In /EEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for vision-language models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022a.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision- language models. International Journal of Computer Vision (ICV), 2022b.
13

---

A Related Work
A.1 Applicability of VP.
VP has been successfully applied to black-box scenarios where model internals are inaccessible. sai al. 2020) proposed black-box adversarial reprogramming (BAR), which leverages zeroth-order optimization, using only numerical evaluations of the loss function instead of gradients. [Oh et al. 2023 further advanced this direction with Black enabling adaptation without knowledge of the recognition model's s architecture or parameters. VP has shown promise in adapting models at test time handle distribution shifts. Gan al. 2023 applied VP test-time adaptation, improving inference accuracy on out-of-distributin data by updating prompts during test time without labels. Tsai et al. introduced convolutional visual prompt (CVP), which are pre-trained through self-supervised learning and adapted at test time to enhance out-of -it-of---istibuuiin performance.
2023
A.2 Visual Prompt Tuning.
Visual prompt tuning (VPT) [Jia et al. 2022 Han al. 2024 2023 Zeng et al. 2024] is conceptually related to visual prompting. VPT adapts to downstream tasks by optimizing input embeddings as well as the classification head of the model. In contrast, VP methods attach learnable parameters directly to the input image, and typically adjust the classification head using heuristic methods [Wu
et al. 2024 Chen et al. 2023 Cai et al. 2024a rather than end-to- and-to-end optimization.
Importantly, VP methods are often designed for settings where the architecture of the recognition model is unknown or the gradients are inaccessible, as in black-box scenarios (Tsai et al. 2020 Oh
et al. 2023
B Theoretical Analysis
In this section, we provide a theoretical analysis of ACAVP within the framework of probably approximately correct (PAC) learning [Kearns and Vazirani 1994]. We establish that proposed method has smaller approximation error compared to existing visual prompting approaches, which explains its superior empirical performance.
Our analysis draws inspiration from the theoretical framework established by Cai et al. 2024b for
analyzing Sample-specift Masks for Visual Reprogramming (SMM). Cai et al. [2024b] demonstrated that using sample-specto masks reduces the approximation error compared to shared masks in visual reprogramming. We extend this insight to analyze the relationship between different visual prompting
methods.
According to Cai et al. 2024b), the approximation error measures how close the best achievable error by a hypothesis space is to the thcoretical minimum error on a distribution. When hypothesis spaces exhibit a subset relation, the larger hypothesis space has a smaller approximation error.
Theorem B.1 (Cai et al. (2024b)). Given an inpul space X, a discrete label space Y, and distribution D over X if there are two hypothesis spaces Fi S [s ( : + Y) and Fg S { X + Y) satisfying Fj C Fz, then we have
Emr apx (F) ErT (Fa
B.1 Hypothesis Spaces of Visual Prompting Methods
We now formalize the hypothesis spaces of the three visual prompting methods under consideration. For fixed pre-trained model fp and fixed output mapping function fout (where f/ =fout fP), we define:
EVP Hypothesis Space. Enhanced Visual Prompting (EVP) resizes the input image by a fixed factor and adds learned pattern 8. Since the resize factor predetermined, a fixed mask M can be prepared in advance. The hypothesis space of EVP is defined as:
(oP((') = {s1(()) = (x((r(x) + M o 6), Vx E X),
(4)
where r. r.(.) is a resizing operation with fixed scale factor s, M the predetermined mask, and the learnable pattern.
14

---

Vu Hypothesis Space. VP extends EVP by making the resizing factor learnable. Since the image size changes during training, the mask is dynamically created according the resize parameter. Its hypothesis space is defined as:
F (fp) = (flf(z) = (p((s()) M 08, X),
(5)
where & is a learnable scaling parameter, M is the mask dynamically generated based on 8, and ò is the learnable pattern.
ACAVP Hypothesis Space. Our proposed ACAVP incorporates affine and color transformation. Its hypothesis space is defined as:
paaa (fpp) (flf(z) f'(Affineez, A) & M0 E X),
(6)
where Affine-- A) represents the affine transformation with learnable affine prompt A g3x3 & E B is the learnable color prompt, and d is the learnable additive prompt.
B.2 Relationship Between Hypothesis Spaces
Following the approach Cai al. (002bb]. we can establish the following relationship between these hypothesis spaces:
Proposition B.2. For any fixed pre-trained model fr and fixed output mapping function feur. we have
FeP(f)) C puutiou (fp) pucopp (fp).
Proof. First, show that Fee (ft)
(f)( Fuuloop [F For any function f FFP((f)) we have
f(x) (x)((()) M
(8)
for some fixed scale factor and learnable pattern 5. We can find a corresponding function in pautovp (f'p) by setting the learnable parameter 8, resulting in
f(z) fx(r(())+++++ M 6)
(9)
with the same 8. Therefore, 1 (ff)) which means Eee (f%) C frauuun
Next, we show that If prautop (f))) S ( For any function f E puatuop ( we have
f(x) (r=(r=)) 6)
(10)
for some learnable parameter and pattern We can construct corresponding function in Facavp (f'p) by:
(f))))(
S
0
0
Setting the affine transformation matrix A
0
0
which performs only scaling by $
0
0
1
Setting the color transformation prompt ē = 1 (all ones)
Using the same additive prompt
With these settings, we have:
Affine(z: (r, A) =ra(x)
(11)
Affine(z, A) = rs(x) o = ra(x)
(12)
Therefore,
f(x) f'(Affineez, A) M 2 fp(ra(0) M O
(13)
This shows that f paaae (fp), which means paulovy (f%) C waawe (fp).
Based on Theorem B.1 and Proposition B.2 we can establish main theoretical result:
15

---

Proposition B.3. For any fixed pre-trained model fP, fixed output mapping function fow, and distribution Dr of the target task, we have
Er (F (F")())))
Dr (poutou (fp)) Er (Facoup (fP))
(14)
This theoretical result, following the framework proposed by Cai et al. 2024b). explains why ACAVP achieves superior performance compared to EVP and AutoVP. P. By expanding the hypothesis space through the introduction of affine and color transformation, ACAVP more effectively approximate the optimal function for the target task.
Similar to the observation in Cai et al. [202bb regarding estimation error, we note that while larger hypothesis space typically reduces approximation error, may increase estimation error due to the added complexity. However, our experimental results show that this potential increase in estimation error well- managed through our training approach, particularly through the use of TrivialAugment for overfitting mitigation.
C Implementation Details
In practice, our ACAVP method applies the transformations the following order: first, the affine transformation is applied to the input image; then, the color transformation applied through element- wise multiplication; finally, the additive noise prompt is applied in the designated regions specified by the mask M.
For initialization, we set the affine transformation parameters to perform a resizing operation, inspired by the success of a previous study [Wu al. 2024 Specifically, initialize the parameters to resize 224x224 image 164x164, setting 0.73 (which corresponds to sigmoid (0.73)) and t= = sh 0. This initialization leverages the effectiveness of resizing operations demonstrated in previous research while providing good starting point for optimization. The color transformation prompt is initialized to = 1, corresponding to no color transformation of the original image. The additive prompt 5 is initialized zero, following the initialization strategy used in [Wu et al., 2024). This initialization strategy ensures that the training begins with a controlled transformation (modest resizing) while maintaining the essential visual information, allowing the model gradually learn more complex and task-specific transformations.
During the training process, all three components of ACAVP-afiin color, and additive transfor- mations- are jointly optimized using stochastic gradient descent to minimize the task task-specific loss function. The TrivialAugment data augmentation is applied to each mini-batch before the prompting transformations, providing the necessary input diversity to prevent overfitting.
D Experimental Setup
D.1 Datasets
We evaluated our method on 12 downstream classification tasks: CIFAR100, CIFAR10 Krizhevsky
et al.
2009]. Flowers 102 [Nilsback and Zisserman 2008). Food101 Bossard et al. 2014), Eu-
roSAT Helber et al.) 2019). SUN397 [Xiao et al. 2010) DTD Cimpoi et al. 2014). UCF 101 [Soomro)
ctt al. 2012]. SVHN [Neteer et al. 2011], OxfordPets Parkhi et al.) 2012 GTSRB Houben et al.
2013 and CLEVR Johnson al. 2017). These datasets encompass diverse visual domains, includ- ing natural scenes, human actions, textures, and fine meee-graind object details, providing a comprehensive evaluation of our method's generalization capability.
For dataset splits, we follow the CoOp protocol [Zhou et al. 2022b For datasets with predefined validation sets, we use them directly. For others, we randomly allocate 10% of the training data for validation and use the remaining 90% for training. We select the checkpoint with the highest validation accuracy for final evaluation. Detailed dataset information is provided in Table
D.2 Model Architectures
We evaluate ACAVP on two representative architectures: CLIP ViT-B/32 [Radford al., 2021 and ImageNet-lk [Russakovsky et al. 2015 pre-trained ResNet50 He et al. 2016]. For CLIP
16

---

Table 9: Detailed dataset information.
indicates that data splits provided by CoOp were used.
Dataset
Train Size
Validation Size
Test Size
Classes
Image Size
CIFARIO0 100
45,000
5,000
10,000
100
32x32
CIFARIO
45,000
5,000
10,000
32x32
Flowers 102
4,093
1,633
2,463
102
128x128
Food
50,500
20,200
30,300
101
128x128
EuroSAT
13,500
5,400
8,100
10
128x128
SUN397!
15,880
3,970
19,850
397
128x122
UCF101
7,639
1,898
3,783
101
128x128
SVHN
65,931
7,326
26,032
32x32
OxfordPets
2,944
736
3,669
37
128x128
DTD
2,820
1.128
1,692
47
128x128
GTSRB
35,288
3,921
12,630
43
32x32
CLEVR 63,000
7,000
15,000
8
480x320
experiments, we use the prompt template "This is a photo of a [LABEL]" for all datasets except CLEVR, where we use "This is photo of [LABEL] objects" to better match its task-specific nature. For the ImageNet pre-trained model, we experiment with two label mapping strategies: random label mapping, which randomly assigns ImageNet classes target dataset classes, and the label mapping method proposed by [Wu et al. 2024).
D.3 Baselines
We compare ACAVP against several baseline methods. Zero-shot inference (ZS) using CLIP serves as our basic baseline without any adaptation. We also compare against standard visual prompt- ing (VP) Bahng et al. 2022 that adds learnable prompts around the input image, Enhanced VP (EVP) [W et al., 2024 that applies fixed resizing before adding prompts, and AutoVP (Tsao et al. 2024] that learns the resize ratio parameter along with the prompt. These baselines represent current state-of-the-ars visual prompting techniques, comprehensively evaluating our method's advantages.
D.4 Training details
In our training setup, used initial learning rate of 40 with Cosine Annealing Learning Rate Scheduler to gradually decrease the learning rate throughout training. The total number of epochs was set to 1000 for all experiments. We employed SGD our optimizer with momentum of 0.9. For most datasets, we used a batch size of 256, with the exception of DTD dataset where we used batch size 64 due its smaller size. For input preprocessing, we first resized all training images to 256x256 pixels, then performed center cropping to 224x224 pixels, followed by normalization. For EVP training, additionally applied horizontal flip as data augmentation technique and implemented gradient normalization to stabilize training. When training Auto with CLIP models, we used gradient clipping via torch nn. waec..nn.uils..lle auiils.clppggre. gaap-_rrra_aleeeeeeeaaaaaaaaeeee value to limit the maximum gradient value to 0.001. Similarly, for ACAVP training, we applied gradient clipping with maximum value of 0.001, and when using CLIP models, we also employed gradient normalization for improved stability.
E
Additional Experiments
E.1 Impact of Data Augmentation on ACAVP
We analyze the effectiveness of different data augmentation techniques when applied ACAVP train- ing. The results of our comparison are presented in Table [0] As shown in the table, all augmentation techniques improved the performance of ACAVP compared training without augmentation.
E.2 Visualization Analysis
We show visualizations of the transformed images generated by different VP methods and the Grad- CAM (Selvaraju et al. 2020) activation maps in Figure For the CLIP ViT model, we observe that
17

---

Table 10: Comparison of data augmentation techniques on ACAVP with CLIP ViT-B/32.
Data Augmentation
CIFARIO
Flowers
95.74
87.55
IPIMix
95.96
90.65
Rand Ragment
96.01 1
88.24 1 101
TrimialAugment
96.27 2.0
89.36 2.119
VP
EVP
Vu
ACAVP
Figure 2: Visualization analysis of different VP methods. The first row shows transformed images on the CIFAR 10 dataset using CLIP VIT-B/32. The second row presents transformed images the CIFARIO dataset using ResNet50. The third row displays Grad-CAM activation maps on the Flowers dataset using ResNet50.
ACAVP applics a minimal affine transformation, appearing similar to the initial resize operation. This conservative transformation is logical considering that CLIP ViT already achieves high zero-shot classification accuracy on the CIFAR10 dataset (88.93% as shown in T When model has strong prior knowledge about the target domain, extensive transformations may be unnecessary even detrimental, and our method adaptively learns this characteristic. contrast, when applied to ResNet50 the CIFARI0 dataset, ACAVP performs more substantial transformations compared to other VP approaches. This is consistent with ResNet50's lower zero-shot classification capabilities on this dataset, necessitating
significant input modifications improve accuracy. The ability of ACAVP to adapt its transformation magnitude based on the backbone model inherent capabilities demonstrates its flexibility across different model architectures. The capacity of ACAVP for more expressive transformations stems from the inclusion of both affine transformation and color trans- formation, which provide richer transformation space compared conventional VP approaches that rely solely on additive noise. This enhanced flexibility allows our method to apply appropriate transformations based on the specific requirements of the dataset and backbone model combination. The Grad-CAM. visualization results reveal another important advantage of our approach. ACAVP produces higher activation values across broader regions of the image compared to other VP methods, indicating more comprehensive utilization of the image information. This suggests that our approach enables the model to consider wider range visual features during classification, rather than focusing limited regions observed with other VP methods.
18

---

Table 11: Classification accuracy comparison on the CIFARIO and Flowers datasets using various models. The parentheses indicate model architecture specifications. indicates results with label mapping proposed by Wu al. 2024].
Model
ResNet101
ViT-L/16
DINOV2(VIT-S114))
SigLIP(WT-SS0400)14))
Average
Method
CIFARIO
Flowers
CIFARIO
Flowers
CIFARIO
Flowers
CIFARIO
Flowers
ZS
57.32
5.16
69.49
4.95
72.35
8.89
95.30
88.96
50,30
VP
67.44
10.72
90.69
13.60
81.63
24.69
96.41
97.00
60.27
EVP
72.86
13.52
94.96
27.49
91.63
10.23
98.76
98.66
63.51
Pu
68.82
13.20
94.75
25.74
94.25
43.85
98.72
97.48
67.10
ACAVP
75.99
17.66
95.38
39.71
93.62
38.49
98.83
98.74
69.80
E.3
Hyperparameters sensitivity
In this section, we investigate the sensitivity of ACAVP to its key hyperparameters. The ACAVP introduces several hyperparameters that control the range of transformations applied to input images. Specifically, we analyze the impact of hyperparameters Rg. Rgh, and Re that restrict the range of affine transformation, and Rg that limits the range of color transformation.
We conducted experiments using the CLIP ViT-B/32 model on the CIFARIO and Flowers datasets. For the affine transformation parameters, we varied the scaling factor X applied to the default values (R, 0.05, Rg Rah 0.1). For the color transformation parameter Rg we tested values ranging from 2.5 to 20.0, with the default value being 6.0. All other training settings remained consistent with our main experimental setup.
The results of our hyperparameter sensitivity analysis are presented in Figure 3 As shown in the figure, the performance of ACAVP exhibits some variation across different hyperparameter values, but the impact is relatively modest. For the CIFAR10 dataset, the accuracy ranges from 95.9% to 96.3% across different affine transformation hyperparameter values, and from 96.0% 96.3% for different color transformation hyperparameter values. Similarly, for the Flowers dataset, the accuracy ranges from 88.0% 89.5% for affine transformation parameters and from 88.5% 89.5% for color transformation parameters.
Importantly, ACAVP consistently outperforms the strongest baselines across all hyperparameter configurations tested. For the CIFARÍO dataset, the second- -bett baseline (EVP) achieves 95.87% accuracy, while ACAVP exceeds this performance regardless of the hyperparameter values selected. Similarly, for the Flowers dataset, the second-best baseline (AutoVP) achieves 82.69% accuracy, which is substantially lower than the worst-performing ACAVP configuration.
These results demonstrate that while ACAVP's performance can be further optimized through hyperparameter tuning, the method is relatively robust to hyperparameter choices. This robustness particularly valuable practical applications, as it suggests that ACAVP can achieve strong performance even without extensive hyperparameter optimization. The consistent performance advantage over bascline methods across all hyperparameter configurations further validates the effectiveness of the proposed approach.
E.4 Experiments with other models
We conducted additional experiments using the other models (SigLIP Zhai et al. 2023), DI- NOv2 [Oquab et al. 2023], ViT-L/16, and ResNet101). All experiments were conducted with a single run. We set the number of training epochs to 200. For learning rate optimization, we evaluated two different learning rates; 40 and 0.1, and reported the results from the higher-performing configuration. The results are presented in Table IT ACAVP achieves the highest accuracy across all architectures except DINOv2, where it achieves the accond-highest performance. All VP methods, including ACAVP, can improve the performance of models with high zero-shot capabilities such as LIILIP. Notably, ACAVP demonstrates strong performance even with tate-offthhe-rt vision encoders.
Next, Table T2 shows the computational overhead (the relative time compared to the inference time of each model) for each model. Since these models have shorter inference times than the CLIP ViT-B/32
19

---

80.5
96.3
Acruney (80
06.2
Accunayy
68.0
96.0
96.9
0.50
0.75
1.00
1.25
1.50
1.15
2,00
2.25
2.50
58.0
0.50
0.75
1.00
1.20
1.50
1.75
2.00
2.25
2.50
(a) AVCAVP accuracy when changing the hyperpa- rameters of the affine transformation the CIFAR1O datasct.
(b) AVCAVP accuracy when changing the hyperpa- rameters of the affine transformation on the Flowers dataset.
Acurncy (%)
Acoraey (50
96.0
5.0
2.5
10.0
12.5
150
17.5
230
2.5
5.0
7.5
10.0
12.5
15.0
17.5
200
(c) AVCAVP accuracy when changing the hyperpa- rameter of the color transformation on the CIFARIO dataset.
(d) AVCAVP accuracy when changing the hyperpa- rameter of the color transformation on the Flowers dataset.
Figure 3: Performance of ACAVP with different hyperparameter values using CLIP ViT-B/32. (a) Accuracy on the CIFARIO dataset when scaling affine transformation hyperparameters (R,, Rs. Righ by factor of X from their default values, with the x-axis showing the value of X. (b) Accuracy the Flowers dataset when scaling affine transformation hyperparameters by a factor of X, with the x-axis showing the value of X. (c) Accuracy the CIFAR10 10 dataset with different values of the color transformation hyperparameter Rg, with the x-axis showing the value R.,. (d) Accuracy on the Flowers dataset with different values of the color transformation hyperparameter Rg, with the x-axis showing the value R.
Table 12: Inference time comparison across different methods using H100 GPU with batch size 500. Relative Time is normalized to the model's 's inference time. The parentheses indicate model architecture specifications.
Method
ResNet101
ViT-L/16
DINOV2(VIT-S/14)
SigLIP(VIT-S0400/1111
VP
0.00
0.01
0.01
0.00
EVP
0.02
0.03
0.04
0.01
AutoVP
1.94
3.38
5.33
1.53
ACAVP
0.18
0.31
0.49
0.14
Coordinator
21.92
38.24
60.33
17.32
used in the main experiments, the relative times ACAVP are slightly higher than those reported in T However, across all models, ACAVP requires less than half the inference time of the model itself. Notably, for the high tethhrrrfommngggggg SigLIP, the relative time is only 14, indicating minimal overhead. While have not yet pursued this direction, further optimization of the implementation could potentially reduce the computational time of VP methods even more.
E.5 Experiments on the ImageNet dataset
The VP method primarily targets users with limited computational resources and is generally evaluated on small to medium-sized datasets, but demonstrating performance on standard benchmarks is crucial. For this purpose, we conducted additional experiments using the ImageNet dataset (Russakovsky al. 2013). We used CL.P ViT-B/16 and trained ACAVP for 10 epochs. The results are presented Table
13
Although the performance gain is modest in this preliminary experiment, demonstrates
20

---

Table 13: Classification accuracy comparison on the ImageNet dataset using CLIP ViT-B/16.
Method
ImageNet
ZS
67,4
ACAVP
67.64
Table 14: Comparison classification accuracy with generative models. We used InstructBLIP for VP training and did not use BLIP2 during training, only during testing.
Model
InstructBLIP
BLIP2
Method
CIFARIO
CIFAR100
CIFARIO
CIFAR100
ZS
88.11
82.41
58.41
60.65
EVP
98.40
85.05
83.20
58.39
TVP
98.78
83.10
85.15
62.80
ACAVP
98.85
88.72
85.32
66.67
that ACAVP can be extended to large-scale datasets like ImageNet. This result highlights the potential flexibility and scalability of our method beyond its original low-resource target scenario.
E.6
Experiments generative models and transferability to unseen models
Generative models are increasingly prevalent in current state-of-the-art technology, making it impor- tant demonstrate that our method effective against such models. We used InstructBLIIP (Dail 2023 as the recognition model for VP training and evaluated both InstructBLIP and BLIP2 [Li et al.
al. 2023 on the CIFARIO and CIFARI00 datasets. BLIP2 was not used during VP training but only during evaluation. During this experiment, we did not access BLIP2's architecture or gradient information; we used only its input-output interface. This setting replicates the typical usage scenario of cloud APIs and demonstrates the applicability of our method in actual black-box environments. The results are presented Table The results demonstrate that all VP methods are indeed effective on generative models like InstructBLIP Notably, ACAVP achieves the highest accuracy, outperform- ing existing methods across both datasets. Interestingly, ACAVP exhibits superior transferability compared to TVP, even though TVP incorporates additional loss functions specifically designed to enhance transferability to models not used during training (BLIP2 this case). This suggests that the expanded transformation space and overfitting mitigation introduced by ACAVP effectively improve VP's generalization performance and transferability.
E.7 Experiments on the object detection and semantic segmentation tasks
We conducted experiments on the object detection and semantic segmentation tasks to evaluate the broader applicability of our approach. For the object detection task, we trained and evaluated an SSD [Liu al.) 2016) model (with ImageNet pre-trained VGG16 (Simonyan and Zisserman 2014) backhone) on PASCAL VOC2007 [Everingham et al. 2015). For the semantic segmentation task, trained and evaluated DeepLaby3 [Chen 2017) model (with ImageNet pre-trained
ResNet50 backbone) on PASCAL VOC2012. We jointly optimized VP and the classification heads
of these models during training. The results are presented in Table 15 ACAVP demonstrates superior performance compared to the baseline VP method across both tasks. These results indicate that ACAVP's enhanced transformation space is effective beyond image classification tasks. While the absolute performance on the object detection task is relatively modest, this can be attributed to experimental constraints imposed by limited time resources, specifically the reduced number of training epochs and the use of only PASCAL VOC2007 for training (whereas standard practice typically involves training on both PASCAL VOC2007 and PASCAL VOC2012 datasets).
F.8
Comparisons with Visual Prompt Tuning
We compare ACAVP and VPT Dia et al.
2022 using the accuracy reported in (w et al. 2024 The results are presented in Table T6 While VPT performs better on some datasets, ACAVP outperforms VPT on the majority of them, demonstrating its competitive performance.
21

---

Table 15: Performance comparison on the object detection and semantic segmentation tasks.
Method
PASCAL VOC2007 mAP
PASCAL VOC2012 mloU
ZS
32.8
62.3
ACAVP
33.0
64.1
Table 16: Classification accuracy comparison of ACAVP and VPT using CLIP ViT-B/32.
Method
CIFARIO
CIFAR100
CLEVR
EuroAAT
Hlowers
Food
SVHN
VPT
95.0
76.6
58.6
61.6
94.6
76.2
84.7
92.1
69.3
86.1
ACAVP
96.27
80,06
77.19
67.43
97.51
89,36
77.86
87.55
61.95
91.1
Limitations
While ACAVP demonstrates superior performance across diverse datasets and model architectures, has certain limitations that should be acknowledged, First, although the computational overhead of ACAVP relatively minimal compared to model inference (only 6% of the model's computational cost), it still requires more computation than simpler VP methods such as VP and EVP. This increased computational requirement might become relevant extremely resource-constrained environments where even small additional costs are significant.
ACAVP's transformation space, while significantly more expressive than conventional VP approaches, still limited to affine, color, and additive transformations. More complex transformations, such as non-linear geometric transformations or advanced image processing operations, could potentially further enhance the expressive power of VP. However, incorporating such transformations would likely increase the computational overhead and potentially introduce additional training challenges.
22

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract are detailed in the introduction part, and further discussed in each section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are discussed in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

Justification:
All assumptions and proof are included in Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Implementation details and experimental settings are provided in Appendix C and Appendix D
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [No]
Justification: Due to the policies the affiliated organization, the code cannot be made public until has been accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurlPS code and data submission guidelines (https
hetpolliiiggiddee/ooooooo http:///ddd//oeSuuubntn ssicnpolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed dataset information is included in Appendix D Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Experiments are run on three seeds, with the standard error being reported in all results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The computational resources (GPU used in the experiment and execution time) are described in Table
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conform with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Since the proposed method in this paper are used to improve the performance of VP in downstream classification tasks, there is no potential societal impact.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release data or models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Creators original owners of code data have been cited.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
28

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: LLM is used only for writing, editing, formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
29


Reviews:
[{"review_id": "EDYS9LHV2o", "reviewer": "Reviewer_wRed", "summary": "The authors propose ACAVP to improve visual prompting by introducing complementary (i.e., affine, color, and additive) transformation operations. They also use TrivialAugment to mitigate overfitting problems in VP. The authors evaluate their model on CLIP and Resnet-50 on 12 image classification datasets as well as robustness to distribution shifts on CIFAR10-C and CIFAR100-C datasets.", "questions": "1. I wonder if the authors have tried tasks other than image classification. There are other vision tasks such as segmentation, object detection, etc. Besides vision tasks, there are multi-modal tasks as well such as VQA.\n2. Since CLIP and ResNet seem a bit outdated, I think the authors can try more modern vision models such as SigLIP, DINOv2 and also other larger vision models to see if the method can scale up. Besides discriminative models, it would be interesting to see if the method can also generalize to generative models such as Qwen.\n3. It would be better if the authors can give a better explanation of why specifically choosing affine/color transformation and not other transformation techniques to improve the expressivity.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 1, "strengths_and_weaknesses": "# Strengths\n1. The paper is well-written and easy to follow.\n2. The method is simple but effective.\n3. The paper is technically sound. The authors make several claims (the method can improve expressivity and mitigate overfitting, and is lightweight) and provide with corresponding experiments (SOTA results on image classification datasets, robustness to distribution shifts, computational efficiency) to support the claims.\n4. The authors provide theoretical analysis along with good empirical performance.\n\n# Weaknesses\n1. The evaluation setting is very limited since the authors only evaluate their model on image classification tasks.\n2. The backbones are also limited since the authors only use CLIP and ResNet which seem a bit outdated give new vision encoders recently.\n3. The method seems lack novelty, since in other areas people have also tried affine/color transformations.", "was_revised": true, "final_justification": "The authors provided additional experiments on more backbones and tasks, especially generative models. It would be better if the authors can include more SOTA models such as qwen and more tasks such as VQA. Therefore, I am updating my score to 4.", "rebuttal": "We sincerely thank the reviewer for the constructive feedback and for recognizing the technical soundness, clarity, and comprehensive experimental validation of our work.\nWe address your valuable concerns below.\n\n# Weaknesses.1 & Questions.1\n> The evaluation setting is very limited since the authors only evaluate their model on image classification tasks.\n\n> I wonder if the authors have tried tasks other than image classification. There are other vision tasks such as segmentation, object detection, etc. Besides vision tasks, there are multi-modal tasks as well such as VQA.\n\nThank you for this valuable feedback.\nWe conducted additional experiments on object detection and semantic segmentation tasks to evaluate the broader applicability of our approach.\nFor the object detection task, we trained and evaluated an SSD model (with ImageNet pre-trained VGG16 backbone) on PASCAL VOC2007.\nFor the semantic segmentation task, we trained and evaluated a DeepLabv3 model (with ImageNet pre-trained ResNet50 backbone) on PASCAL VOC2012.\nWe jointly optimized VP and the classification heads of these models during training.\nThe results are presented in the table below.\n\n| Method | PASCAL VOC2007 mAP | PASCAL VOC2012 mIoU |\n|--------|--------------------|---------------------|\n| VP     | 32.8               | 62.3                |\n| ACAVP  | **33.0**           | **64.1**            |\n\nACAVP demonstrates superior performance compared to the baseline VP method across both tasks.\nThese results indicate that ACAVP's enhanced transformation space is effective beyond image classification tasks.\nWhile the absolute performance on the object detection task is relatively modest, this can be attributed to experimental constraints imposed by limited time resources, specifically the reduced number of training epochs and the use of only PASCAL VOC2007 for training (whereas standard practice typically involves training on both PASCAL VOC2007 and PASCAL VOC2012 datasets).\n\n# Weaknesses.2 & Questions.2 \n> The backbones are also limited since the authors only use CLIP and ResNet which seem a bit outdated give new vision encoders recently.\n\n> Since CLIP and ResNet seem a bit outdated, I think the authors can try more modern vision models such as SigLIP, DINOv2 and also other larger vision models to see if the method can scale up. Besides discriminative models, it would be interesting to see if the method can also generalize to generative models such as Qwen.\n\nThank you for your suggestion.\nWe conducted additional experiments using the modern vision models you suggested (SigLIP, DINOv2) as well as ViT-L/16 and ResNet101 recommended by reviewer 63vY (we used ViT-L/16 instead of ViT-L/14 as the latter is not available in torchvision.models).\nThe results are presented in the table below.\n\n| Model  | RN101†    |           | ViT-L/16† |           | DINOv2(ViT-S/14)† |           | SigLIP(ViT-SO400M/14) |           | Average     |\n|--------|-----------|-----------|-----------|-----------|-------------------|-----------|-----------------------|-----------|-------------|\n| Method | CIFAR10   | Flowers   | CIFAR10   | Flowers   | CIFAR10           | Flowers   | CIFAR10               | Flowers   |             |\n| ZS     | 57.32     | 5.156     | 69.49     | 4.953     | 72.35             | 8.892     | 95.3                  | 88.957    | 50.30225    |\n| VP     | 67.44     | 10.72     | 90.69     | 13.6      | 81.63             | 24.69     | 96.41                 | 97        | 60.2725     |\n| EVP    | 72.86     | 13.52     | 94.96     | 27.49     | 91.63             | 10.23     | 98.76                 | 98.66     | 63.51375    |\n| AutoVP | 68.82     | 13.2      | 94.75     | 25.74     | **94.25**         | **43.85** | 98.72                 | 97.48     | 67.10125    |\n| ACAVP  | **75.99** | **17.66** | **95.38** | **39.71** | 93.62             | 38.49     | **98.83**             | **98.74** | **69.8025** |\n\n† indicates the use of output mapping proposed in EVP.\nACAVP achieves the highest accuracy across all architectures except DINOv2, where it achieves the second-highest performance.\nAll VP methods, including ACAVP, can improve the performance of models with high zero-shot capabilities such as SigLIP.\nNotably, ACAVP demonstrates strong performance even with state-of-the-art vision encoders.\n\nRegarding experiments with generative models such as Qwen, we were unable to complete these within the limited rebuttal timeframe.\nRecent work by Zhang et al. [1] has demonstrated that visual prompting can be successfully applied to multimodal large language models, including generative models like MiniGPT-4, showing consistent improvements across various vision-language tasks.\nGiven that ACAVP extends the transformation space of traditional VP through affine and color transformations, we believe that ACAVP would similarly benefit generative models.\n\n[1] Zhang, Yichi, et al. \"Exploring the transferability of visual prompting for multimodal large language models.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n# Weaknesses.3 & Questions.3\n> The method seems lack novelty, since in other areas people have also tried affine/color transformations.\n\n> It would be better if the authors can give a better explanation of why specifically choosing affine/color transformation and not other transformation techniques to improve the expressivity.\n\nThank you for raising this important point about novelty and transformation choice.\nWe acknowledge that affine and color transformations are well-established techniques in computer vision. However, our contribution lies not in the individual techniques themselves, but in their strategic integration into the visual prompting paradigm and the systematic analysis of their effectiveness in this specific context.\nOur choice of affine and color transformations was motivated by several key considerations:\n(1) These transformations are differentiable and can be seamlessly integrated into gradient-based optimization, unlike many other transformation techniques.\n(2)They provide fundamentally different transformation capabilities compared to traditional additive noise in VP—affine transformations modify spatial relationships while color transformations adjust feature intensities, creating a more expressive transformation space.\n(3) Both transformations maintain VP's core advantage of minimal inference overhead, unlike more complex alternatives such as neural network-based transformations.\n(4) As shown in our analysis (Appendix B), this combination provably reduces approximation error compared to existing VP methods.\n\nWhile the individual transformations are not novel, their principled combination, theoretical analysis, and empirical validation in the VP context represent our key contributions. This approach has enabled ACAVP to achieve state-of-the-art performance while maintaining the fundamental advantages of visual prompting."}, {"review_id": "N4DRQ0ad2t", "reviewer": "Reviewer_kCxW", "summary": "This paper proposes a novel visual prompting method, ACAVP, which leverages learnable affine transformations, color adjustments, and edge pixel prompts on the input image to achieve finetuning of black-box models. Experimental results demonstrate the superiority of this method over existing visual prompting approaches.", "questions": "See above", "limitations": "N/A", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths \n+ The paper proposes a simple yet effective method, and the experimental results are convincing.\n+ The paper also explores the overfitting issues in existing VP methods and provides an effective solution.\n\nWeaknesses\n+ My main concern lies in the value of VP as a general methodology. From the experimental results in the paper, it is evident that VP works well for models like CLIP, which already perform strongly in zero-shot settings. However, for models pre-trained on ImageNet, the performance gap between VP and full finetuning is substantial, rendering VP far from usable in practice. This suggests that the capabilities of VP may be limited to making small adjustments to already strong models. For models that do not support zero-shot inference to begin with, VP may not be a promising direction.", "was_revised": true, "final_justification": "The author resolved my issue, and I will keep my rating.", "rebuttal": "We thank the reviewer for this constructive review and for recognizing the effectiveness of our method and our contribution in addressing overfitting issues in visual prompting.\nWe address your concerns below.\n\n# Weaknesses\n> My main concern lies in the value of VP as a general methodology. From the experimental results in the paper, it is evident that VP works well for models like CLIP, which already perform strongly in zero-shot settings. However, for models pre-trained on ImageNet, the performance gap between VP and full finetuning is substantial, rendering VP far from usable in practice. This suggests that the capabilities of VP may be limited to making small adjustments to already strong models. For models that do not support zero-shot inference to begin with, VP may not be a promising direction.\n\nWe appreciate this insightful observation about VP's performance characteristics across different model types.\n\nWe acknowledge that VP indeed achieves higher accuracy when applied to models with strong zero-shot capabilities like CLIP compared to ImageNet pre-trained models, as you correctly point out.\nHowever, we believe this does not diminish VP's practical utility for several reasons.\n\nMost importantly, in current practice, CLIP and similar vision-language models have become the predominant choice for vision tasks due to their superior zero-shot capabilities and versatility.\nWhile CLIP models have slightly higher inference costs due to text encoding, their significantly superior performance and zero-shot capabilities make them the preferred choice for practical applications.\nFrom this perspective, the performance gap on ImageNet pre-trained models is less concerning in practice, as practitioners typically prefer to fine-tune stronger foundation models like CLIP rather than ImageNet pre-trained models.\n\nFurthermore, our results demonstrate that VP is not limited to small adjustments to already strong models. For instance, on specialized datasets like GTSRB and SVHN, CLIP's zero-shot performance is quite low (18.18% and 15.30% respectively), yet VP methods achieve substantial improvements to over 90% accuracy (92.04% and 91.81% with ACAVP). These dramatic performance gains—improvements of over 70 percentage points—clearly show that VP can enable significant task adaptation rather than merely fine-tuning already capable models.\n\nAdditionally, VP possesses a unique characteristic that distinguishes it from other parameter-efficient fine-tuning methods: its applicability to black-box models where the architecture is unknown and gradients are inaccessible.\nThis capability enables VP to adapt state-of-the-art multimodal large language models provided as APIs to downstream tasks, which is impossible with traditional fine-tuning approaches.\n\nWhile we included ImageNet pre-trained model experiments for comprehensive evaluation, we believe the practical significance lies in VP's performance on modern foundation models like CLIP, where our method demonstrates clear advantages.\nTherefore, we believe VP represents a promising research direction that addresses scenarios where conventional adaptation methods are not applicable, particularly in the era of increasingly powerful foundation models distributed through API services."}, {"review_id": "26E52NYriY", "reviewer": "Reviewer_63vY", "summary": "This study addresses two fundamental limitations in visual prompting (VP)—namely, restricted expressive power and pronounced overfitting tendencies—by introducing ACAVP (Affine, Color, and Additive Visual Prompting), a multimodal visual prompting framework that integrates affine transformations, color transformations, and additive perturbations.​This approach fundamentally expands VP's transformation space by dynamically generating task-adaptive prompt regions through affine transformations (preserving original image information) while enhancing discriminative features via color transformations (adjusting brightness/contrast). Crucially, it systematically identifies TrivialAugment data augmentation as the optimal overfitting mitigation strategy. Extensive experimentation across 12 datasets and two architectures (ViT/ResNet50) demonstrates that ACAVP achieves state-of-the-art performance: attaining 83.18% average accuracy on CLIP-ViT (surpassing all existing VP methods and linear probing), improving robustness by 7.33% on distribution-shifted CIFAR10-C data, and maintaining VP's computational efficiency with merely 6% inference overhead. This work systematically resolves the fundamental trade-off between expressive power and generalization capability.", "questions": "Question 1\n\nThe paper claims ACAVP enhances visual prompt expressiveness via affine and color transformations but does not rigorously demonstrate the complementarity of these operations. How do the authors rule out redundancy—i.e., whether a single transformation alone can achieve similar or better performance?\n\nQuestion 2\n\nDoes the distribution of zero-value pixels after affine transformation exhibit task adaptability? As shown in Supplementary Figure 2, affine transformation magnitudes vary significantly between datasets (CIFAR10 vs. Flowers)—does this lead to instability in masked regions?\n\nQuestion 3\n\nWhy was TrivialAugment selected as the default configuration? Is its computational efficiency better aligned with the lightweight design principle of visual prompts (VPs)?", "limitations": "Experimental validation is inadequate and needs extension. Specifically, the study lacks comparative evaluations of more diverse backbone architectures and fails to benchmark on a broader range of mainstream datasets.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1.Proposes a tri-modal visual prompt framework (ACAVP) fusing affine, color, and additive transformations, which systematically addresses the intrinsic limitation of traditional visual prompt (VP) methods—their exclusive reliance on additive noise as the sole prompt form.\n2.Uncovers general patterns of overfitting in VP tasks and demonstrates the universal utility of TrivialAugment for improving VP generalization, providing a critical regularization paradigm for the field.\n\n Weaknesses：\n\n1.The experiments are solely conducted on two architectures CLIP ViT-B/32 and ResNet50，and computational overhead (e.g., inference time, GPU memory consumption) is only reported for CLIP ViT-B/32, lacking comparative analysis across different backbones. To support claims of generality, overhead data for additional architectures (e.g., ViT-L/14, ResNet101) should be supplemented, along with an analysis of how overhead scales with architecture complexity.\n\n2.The ablation experiments for the three core components are incompleteFurthermore, the backbone architecture used in these ablations is not explicitly stated.  Comprehensive ablation studies including full combinations should be expanded, and the backbone corresponding to each ablation explicitly labeled to ensure conclusion traceability.\n\n3.Experiments in Tables 5 and 6 are validated on only two datasets, providing insufficient coverage to fully demonstrate the method’s generalization ability across diverse data distributions. Most critically, results on canonical benchmarks like ImageNet are absent.", "was_revised": true, "final_justification": "The authors’ rebuttal and subsequent discussions have addressed most of the concerns and provided additional clarifications and experiments regarding black-box model settings (such as cloud APIs) and larger-scale datasets. Therefore, I have decided to raise my score.", "rebuttal": "We thank the reviewer for this thorough review and for recognizing our systematic approach to addressing VP's limitations through the tri-modal framework and our identification of overfitting patterns with effective regularization solutions.\nWe address your concerns below.\n\n# Weaknesses.1\n> The experiments are solely conducted on two architectures CLIP ViT-B/32 and ResNet50，and computational overhead (e.g., inference time, GPU memory consumption) is only reported for CLIP ViT-B/32, lacking comparative analysis across different backbones. To support claims of generality, overhead data for additional architectures (e.g., ViT-L/14, ResNet101) should be supplemented, along with an analysis of how overhead scales with architecture complexity.\n\nThank you for this important feedback.\nWe conducted additional experiments using ViT-L/16 and ResNet101 (we used ViT-L/16 instead of ViT-L/14 as the latter is not available in torchvision.models) as you suggested, as well as SigLIP and DINOv2 as requested by reviewer wRed.\nThe accuracy results are shown in the table in our response to reviewer wRed, and the computational overhead (relative time compared to each model's inference time) is presented in the table below.\n\n| Method      | RN101 | ViT-L/16 | DINOv2(ViT-S/14) | SigLIP(ViT-SO400M/14) |\n|-------------|-------|----------|------------------|-----------------------|\n| VP          | 0.00  | 0.01     | 0.01             | 0.00                  |\n| EVP         | 0.02  | 0.03     | 0.04             | 0.01                  |\n| AutoVP      | 1.94  | 3.38     | 5.33             | 1.53                  |\n| ACAVP       | 0.18  | 0.31     | 0.49             | 0.14                  |\n| Coordinator | 21.92 | 38.24    | 60.33            | 17.32                 |\n\n\nSince these models have shorter inference times than the CLIP ViT-B/32 used in our paper, the relative times for ACAVP are slightly higher than those reported in Table 8.\nHowever, across all models, ACAVP requires less than half the inference time of the model itself.\nNotably, for the high-performing SigLIP, the relative time is only 0.14, indicating minimal overhead.\nWhile we have not yet pursued this direction, further optimization of the implementation could potentially reduce the computational time of VP methods even more.\n\n# Weaknesses.2 & Questions.1\n> The ablation experiments for the three core components are incompleteFurthermore, the backbone architecture used in these ablations is not explicitly stated. Comprehensive ablation studies including full combinations should be expanded, and the backbone corresponding to each ablation explicitly labeled to ensure conclusion traceability.\n\n> The paper claims ACAVP enhances visual prompt expressiveness via affine and color transformations but does not rigorously demonstrate the complementarity of these operations. How do the authors rule out redundancy—i.e., whether a single transformation alone can achieve similar or better performance?\n\nThank you for pointing out the missing specification of the model used in our ablation studies.\nWe used CLIP ViT-B/32 for all ablation experiments.\nWhile we measured the impact of adding Affine and Color transformations to EVP in our paper, we had not evaluated the individual effects of Affine and Color transformations alone.\nWe conducted additional comprehensive ablation studies to address this concern.\nThe results are shown in the table below.\n\n| Transformation                    | CIFAR10      | Flowers      |\n|-----------------------------------|--------------|--------------|\n| None (ZS)                         | 88.93        | 61.63        |\n| Affine                            | 90.17 ± 0.01 | 57.42 ± 0.03 |\n| Color                             | 92.14 ± 0.06 | 62.92 ± 0.04 |\n| Affine + Color                    | 92.46 ± 0.04 | 60.58 ± 0.08 |\n| Affine + Color + Additive (ACAVP) | 96.27 ± 0.07 | 89.36 ± 0.19 |\n\nIndividual transformations show varying effectiveness across datasets: Affine transformation improves performance on CIFAR10 but shows reduced performance on Flowers compared to zero-shot, while Color transformation consistently outperforms zero-shot accuracy on both datasets.\nCombining Affine and Color transformations yields further improvements.\nFurthermore, adding the traditional Additive transformation achieves the highest accuracy.\nThese results demonstrate that Affine, Color, and Additive transformations have complementary relationships, and the use of these transformations is not redundant.\n\n# Weaknesses.3\n> Experiments in Tables 5 and 6 are validated on only two datasets, providing insufficient coverage to fully demonstrate the method’s generalization ability across diverse data distributions. \n\nWe appreciate your valuable feedback. Due to limited computational resources, we were unable to conduct the detailed analysis (Tables 5 and 6) across all 12 datasets used in the main experiments. \nThese supplementary experiments were designed to provide deeper insights into the internal workings of ACAVP, while the main experimental results (Tables 2 and 3) already verify its generalization capability across a diverse set of tasks. \nNonetheless, we agree that additional dataset coverage would further strengthen our claims. \nWe will work to expand these analyses to all 12 datasets in the camera-ready version.\n\n> Most critically, results on canonical benchmarks like ImageNet are absent.\n\nThank you for pointing this out. While VP methods, including ours, are primarily aimed at users with limited resources and are generally evaluated on small- to medium-scale datasets, we understand the importance of demonstrating performance on canonical benchmarks. \nTo that end, we performed an additional experiment on ImageNet. \nAs shown below, ACAVP achieved slightly better performance than the zero-shot baseline, using CLIP ViT-B/16 fine-tuned for 10 epochs:\n\n| Method | Accuracy  |\n|--------|-----------|\n| ZS     | 67.48     |\n| ACAVP  | **67.64** |\n\nAlthough the performance gain is modest in this preliminary experiment, it demonstrates that ACAVP can be extended to large-scale datasets like ImageNet. This result highlights the potential flexibility and scalability of our method beyond its original low-resource target scenario.\n\n\n# Questions.2\n> Does the distribution of zero-value pixels after affine transformation exhibit task adaptability? As shown in Supplementary Figure 2, affine transformation magnitudes vary significantly between datasets (CIFAR10 vs. Flowers)—\n\nYes, the distribution of zero-value pixels after affine transformation does exhibit task adaptability. \n\nOur dynamic mask generation mechanism (M) automatically adapts to the zero-value pixel distribution created by the learned affine transformation for each task. When the affine transformation creates larger empty regions (as in Flowers), more pixels become available for additive prompting, allowing for greater input space modification. Conversely, when smaller transformations are sufficient (as in CIFAR10), fewer zero-value pixels are created, resulting in more focused prompt application.\n\nThis adaptive behavior reflects the varying adaptation requirements across tasks:\n- **Distribution-similar tasks** (ImageNet→CIFAR10): Minimal affine transformation creates small zero-value regions, indicating that subtle spatial adjustments with limited additive prompting suffice\n- **Distribution-different tasks** (ImageNet→Flowers): Larger affine transformations create extensive zero-value regions, enabling more substantial input modifications through expanded additive prompting areas\n\n> does this lead to instability in masked regions?\n\nYes, the masked regions are inherently unstable as they vary across different training runs even with the same task and model due to the stochastic nature of the learning process.\nHowever, this instability in masked regions does not adversely affect task performance.\nAccording to the results in Tables 2 and 3, ACAVP exhibits small standard errors across most experimental settings, demonstrating that recognition performance remains stable despite the variability in mask generation.\nThe key insight is that while individual mask patterns may vary between training runs, the overall adaptation capability and performance remain consistent, indicating that our method learns robust task-specific transformations regardless of the specific mask configuration.\n\n# Questions.3\n> Why was TrivialAugment selected as the default configuration? Is its computational efficiency better aligned with the lightweight design principle of visual prompts (VPs)?\n\nThank you for the thoughtful question.\nTo ensure practical usability without the overhead of policy search, we considered three representative data augmentation methods that are lightweight and easy to apply. As shown in Table 6, TrivialAugment yielded the best performance among them, and was thus selected as the default configuration.\n\nWhile TrivialAugment introduces some additional cost during training, it has no impact on inference-time computation. Since the key design principle of VPs lies in minimizing inference-time overhead, the use of TrivialAugment preserves the lightweight nature of our method."}, {"review_id": "ANyq2gYkxH", "reviewer": "Reviewer_Aezh", "summary": "While conventional VP methods achieve lower accuracy than other adaptation approaches, the paper proposes ACAVP (Affine, Color, and Additive Visual Prompting), which enhances VP’s expressive power by introducing complementary transformation operations. Here affine transformation can create task-specific prompt regions while preserving original image information, and color transformation, on the other hand, can emphasize task-relevant visual features.", "questions": "Please see my above concern.", "limitations": "N/A", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. The paper is easy to follow. By introducing the affine transformation for creating task-specific prompt regions, this approach is able to robustly improve VP's performance.\n\nWeakness:\n1. A missing area of this paper is visual prompt tuning/prefix tuning in language [1-4], which is very similar to visual prompting techniques. The authors need to clearly separate the difference between VP and VPT. Also, as stated in [5], visual prompt tuning seems to be competitive with full fine-tuning under some conditions. Does the claim \"VP methods typically achieve lower accuracy than full fine-tuning\" logically correct?\n2. Similar to [6], the authors may need to consider involving other competitive PEFT baselines for completeness (e.g., prompt tuning, adapter)\n\n[1] Visual prompt tuning\n\n[2] E2VPT: An Effective and Efficient Approach for Visual Prompt Tuning\n\n[3] Visual Fourier Prompt Tuning\n\n[4] Aprompt: Attention prompt tuning for efficient adaptation of pre-trained language models\n\n[5] Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?\n\n[6] BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for this thorough review and for recognizing the clarity of our paper and the effectiveness of the proposed affine transformation in improving VP performance.\nWe address your concerns below.\n\n# Weaknesses.1\n> A missing area of this paper is visual prompt tuning/prefix tuning in language [1-4], which is very similar to visual prompting techniques. The authors need to clearly separate the difference between VP and VPT. \n\nThank you for the insightful comment.\nWe agree that visual prompt tuning and prefix tuning in language domains are conceptually related to visual prompting (VP). \nAs described in [1–4], VPT adapts to downstream tasks by optimizing input embeddings as well as the classification head of the model. \nIn contrast, VP methods attach learnable parameters directly to the input image, and typically adjust the classification head using heuristic methods [7, 8, 9], rather than end-to-end optimization.\n\nImportantly, VP methods are often designed for settings where the architecture of the recognition model is unknown or the gradients are inaccessible, as in black-box scenarios [6].\nWe will revise the paper to more clearly articulate these differences between VP and VPT, and to appropriately position our work in this context.\n\n[7]  Wu, Junyang, et al. \"Unleashing the power of visual prompting at the pixel level.\" TMLR. 2024.\n\n[8] Chen, Aochuan, et al. \"Understanding and improving visual prompting: A label-mapping perspective.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[9] Cai, Chengyi, et al. \"Bayesian-guided label mapping for visual reprogramming.\" The Thirty-eighth Annual Conference on Neural Information Processing Systems. 2024.\n\n\n> Also, as stated in [5], visual prompt tuning seems to be competitive with full fine-tuning under some conditions. Does the claim \"VP methods typically achieve lower accuracy than full fine-tuning\" logically correct?\n\nThank you for highlighting this interesting point.\nAs shown in [5], when the pretraining and downstream datasets share similar distributions, VPT can outperform full fine-tuning (FT). \nThis is consistent with our own results (Table 2), where ACAVP achieves higher accuracy than FT on datasets such as CIFAR-10 and SUN.\nWe agree that the original statement \"VP methods typically achieve lower accuracy than full fine-tuning\" is somewhat of an overclaim. \nA more accurate description would be \"VP methods often achieve lower accuracy than full fine-tuning.\"\nWe will revise the wording accordingly in the camera-ready version.\n\n\n# Weaknesses.2\n> Similar to [6], the authors may need to consider involving other competitive PEFT baselines for completeness (e.g., prompt tuning, adapter)\n\nThank you for pointing this out. While [6] does not include empirical accuracy comparisons with PEFT methods such as VPT or prompt tuning, it does provide a taxonomy comparing their settings (e.g., prompt location, gradient access, and input dependency).\nOur method is an extension of VP and thus shares the same setting as VP and BlackVIP. \nNevertheless, to further clarify the relationship between VP-based methods and other PEFT approaches such as VPT and adapter tuning, we will explicitly describe these differences in the related work section of the camera-ready version.\n\nAdditionally, we include below a comparison of ACAVP and VPT using the accuracy reported in [7]:\n\n\n| Method | CIFAR10          | CIFAR100         | CLEVR            | DTD              | EuroSAT          | Flowers          | Food         | Pets         | SUN          | SVHN             |\n|--------|------------------|------------------|------------------|------------------|------------------|------------------|--------------|--------------|--------------|------------------|\n| VPT    | 95               | 76.6             | 58.6             | 61.6             | 94.6             | 76.2             | **84.7**     | **92.1**     | **69.3**     | 86.1             |\n| ACAVP  | **96.27 ± 0.07** | **80.06 ± 0.25** | **77.19 ± 0.17** | **67.43 ± 0.83** | **97.51 ± 0.09** | **89.36 ± 0.19** | 77.86 ± 0.09 | 87.55 ± 0.02 | 64.95 ± 0.10 | **91.81 ± 0.18** |\n\n\nWhile VPT performs better on some datasets, ACAVP outperforms VPT on the majority of them, demonstrating its competitive performance."}]

Author rebuttal:

Meta-review: This paper aims to enhance visual prompting. Specifically, it contains three main components: an affine transformation to create task-specific prompt regions, a color transformation to amplify salient features, and TrivialAugment for data augmentation. Empirical results are provided across a range of benchmarks to support its effectiveness. 

Overall, the reviewers agreed that the strategy is clearly motivated and supported by strong empirical results. However, they also raised several concerns: 1) more evaluations beyond image classification should be included; 2) other competitive PEFT baselines should be considered; 3) more comprehrensive ablations are needed, such as isolating the performance gains from the affine transformation alone and testing under black-box settings; and 4) the novelty contribution should be more carefully and clearly articulated, as each individual component itself is not technical novel.

The rebuttal is considered, which addresses most of these concerns. As the result, all reviewers uniamously agree to accept this submission. For the final version, please ensure that all promised revisions from the rebuttal are carefully incorporated to further strengthen the paper.
Decision: Accept (poster)