Title: Enhancing Visual Prompting through Expanded Transformation Space and Overfitting Mitigation
Venue: NeurIPS 2025
Accepted: yes
Keywords: visual prompting

adversarial reprogramming

fine tuning
Abstract: Visual prompting (VP) has emerged as a promising parameter-efficient fine-tuning approach for adapting pre-trained vision models to downstream tasks without modifying model parameters. Despite offering advantages like negligible computational overhead and compatibility with black-box models, conventional VP methods typically achieve lower accuracy than other adaptation approaches. Our analysis reveals two critical limitations: the restricted expressivity of simple additive transformation and a tendency toward overfitting when the parameter count increases. To address these challenges, we propose ACAVP (Affine, Color, and Additive Visual Prompting), which enhances VP's expressive power by introducing complementary transformation operations: affine transformation for creating task-specific prompt regions while preserving original image information, and color transformation for emphasizing task-relevant visual features. Additionally, we identify that overfitting is a critical issue in VP training and introduce TrivialAugment as an effective data augmentation, which not only benefits our approach but also significantly improves existing VP methods, with performance gains of up to 12 percentage points on certain datasets. This demonstrates that appropriate data augmentation is universally beneficial for VP training. Extensive experiments across twelve diverse image classification datasets with two different model architectures demonstrate that ACAVP achieves state-of-the-art accuracy among VP methods, surpasses linear probing in average accuracy, and exhibits superior robustness to distribution shifts, all while maintaining minimal computational overhead during inference.
Our code is available at https://github.com/s-enmt/ACAVP.


Peer reviews:
comments: SUMMARY:
While conventional VP methods achieve lower accuracy than other adaptation approaches, the paper proposes ACAVP (Affine, Color, and Additive Visual Prompting), which enhances VP’s expressive power by introducing complementary transformation operations. Here affine transformation can create task-specific prompt regions while preserving original image information, and color transformation, on the other hand, can emphasize task-relevant visual features.

STRENGTHS AND WEAKNESSES:
Strength:
1. The paper is easy to follow. By introducing the affine transformation for creating task-specific prompt regions, this approach is able to robustly improve VP's performance.

Weakness:
1. A missing area of this paper is visual prompt tuning/prefix tuning in language [1-4], which is very similar to visual prompting techniques. The authors need to clearly separate the difference between VP and VPT. Also, as stated in [5], visual prompt tuning seems to be competitive with full fine-tuning under some conditions. Does the claim "VP methods typically achieve lower accuracy than full fine-tuning" logically correct?
2. Similar to [6], the authors may need to consider involving other competitive PEFT baselines for completeness (e.g., prompt tuning, adapter)

[1] Visual prompt tuning

[2] E2VPT: An Effective and Efficient Approach for Visual Prompt Tuning

[3] Visual Fourier Prompt Tuning

[4] Aprompt: Attention prompt tuning for efficient adaptation of pre-trained language models

[5] Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?

[6] BlackVIP: Black-Box Visual Prompting for Robust Transfer Learning

QUESTIONS:
Please see my above concern.

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study addresses two fundamental limitations in visual prompting (VP)—namely, restricted expressive power and pronounced overfitting tendencies—by introducing ACAVP (Affine, Color, and Additive Visual Prompting), a multimodal visual prompting framework that integrates affine transformations, color transformations, and additive perturbations.​This approach fundamentally expands VP's transformation space by dynamically generating task-adaptive prompt regions through affine transformations (preserving original image information) while enhancing discriminative features via color transformations (adjusting brightness/contrast). Crucially, it systematically identifies TrivialAugment data augmentation as the optimal overfitting mitigation strategy. Extensive experimentation across 12 datasets and two architectures (ViT/ResNet50) demonstrates that ACAVP achieves state-of-the-art performance: attaining 83.18% average accuracy on CLIP-ViT (surpassing all existing VP methods and linear probing), improving robustness by 7.33% on distribution-shifted CIFAR10-C data, and maintaining VP's computational efficiency with merely 6% inference overhead. This work systematically resolves the fundamental trade-off between expressive power and generalization capability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1.Proposes a tri-modal visual prompt framework (ACAVP) fusing affine, color, and additive transformations, which systematically addresses the intrinsic limitation of traditional visual prompt (VP) methods—their exclusive reliance on additive noise as the sole prompt form.\n2.Uncovers general patterns of overfitting in VP tasks and demonstrates the universal utility of TrivialAugment for improving VP generalization, providing a critical regularization paradigm for the field.\n\n Weaknesses：\n\n1.The experiments are solely conducted on two architectures CLIP ViT-B/32 and ResNet50，and computational overhead (e.g., inference time, GPU memory consumption) is only reported for CLIP ViT-B/32, lacking comparative analysis across different backbones. To support claims of generality, overhead data for additional architectures (e.g., ViT-L/14, ResNet101) should be supplemented, along with an analysis of how overhead scales with architecture complexity.\n\n2.The ablation experiments for the three core components are incompleteFurthermore, the backbone architecture used in these ablations is not explicitly stated.  Comprehensive ablation studies including full combinations should be expanded, and the backbone corresponding to each ablation explicitly labeled to ensure conclusion traceability.\n\n3.Experiments in Tables 5 and 6 are validated on only two datasets, providing insufficient coverage to fully demonstrate the method’s generalization ability across diverse data distributions. Most critically, results on canonical benchmarks like ImageNet are absent.\n\nQUESTIONS:\nQuestion 1\n\nThe paper claims ACAVP enhances visual prompt expressiveness via affine and color transformations but does not rigorously demonstrate the complementarity of these operations. How do the authors rule out redundancy—i.e., whether a single transformation alone can achieve similar or better performance?\n\nQuestion 2\n\nDoes the distribution of zero-value pixels after affine transformation exhibit task adaptability? As shown in Supplementary Figure 2, affine transformation magnitudes vary significantly between datasets (CIFAR10 vs. Flowers)—does this lead to instability in masked regions?\n\nQuestion 3\n\nWhy was TrivialAugment selected as the default configuration? Is its computational efficiency better aligned with the lightweight design principle of visual prompts (VPs)?\n\nLIMITATIONS:\nExperimental validation is inadequate and needs extension. Specifically, the study lacks comparative evaluations of more diverse backbone architectures and fails to benchmark on a broader range of mainstream datasets.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

comments: SUMMARY:
This paper proposes a novel visual prompting method, ACAVP, which leverages learnable affine transformations, color adjustments, and edge pixel prompts on the input image to achieve finetuning of black-box models. Experimental results demonstrate the superiority of this method over existing visual prompting approaches.

STRENGTHS AND WEAKNESSES:
Strengths 
+ The paper proposes a simple yet effective method, and the experimental results are convincing.
+ The paper also explores the overfitting issues in existing VP methods and provides an effective solution.

Weaknesses
+ My main concern lies in the value of VP as a general methodology. From the experimental results in the paper, it is evident that VP works well for models like CLIP, which already perform strongly in zero-shot settings. However, for models pre-trained on ImageNet, the performance gap between VP and full finetuning is substantial, rendering VP far from usable in practice. This suggests that the capabilities of VP may be limited to making small adjustments to already strong models. For models that do not support zero-shot inference to begin with, VP may not be a promising direction.

QUESTIONS:
See above

LIMITATIONS:
N/A
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose ACAVP to improve visual prompting by introducing complementary (i.e., affine, color, and additive) transformation operations. They also use TrivialAugment to mitigate overfitting problems in VP. The authors evaluate their model on CLIP and Resnet-50 on 12 image classification datasets as well as robustness to distribution shifts on CIFAR10-C and CIFAR100-C datasets.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n1. The paper is well-written and easy to follow.\n2. The method is simple but effective.\n3. The paper is technically sound. The authors make several claims (the method can improve expressivity and mitigate overfitting, and is lightweight) and provide with corresponding experiments (SOTA results on image classification datasets, robustness to distribution shifts, computational efficiency) to support the claims.\n4. The authors provide theoretical analysis along with good empirical performance.\n\n# Weaknesses\n1. The evaluation setting is very limited since the authors only evaluate their model on image classification tasks.\n2. The backbones are also limited since the authors only use CLIP and ResNet which seem a bit outdated give new vision encoders recently.\n3. The method seems lack novelty, since in other areas people have also tried affine/color transformations.\n\nQUESTIONS:\n1. I wonder if the authors have tried tasks other than image classification. There are other vision tasks such as segmentation, object detection, etc. Besides vision tasks, there are multi-modal tasks as well such as VQA.\n2. Since CLIP and ResNet seem a bit outdated, I think the authors can try more modern vision models such as SigLIP, DINOv2 and also other larger vision models to see if the method can scale up. Besides discriminative models, it would be interesting to see if the method can also generalize to generative models such as Qwen.\n3. It would be better if the authors can give a better explanation of why specifically choosing affine/color transformation and not other transformation techniques to improve the expressivity.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 1, "DATE": "2025-07-02"}