LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptation
Authors: Can Jin
Ying Li
Mingyu Zhao
Shiyu Zhao
Zhenting Wang
Xiaoxiao He
Ligong Han
Tong Che
Dimitris N. Metaxas
Venue: ICLR 2025 Poster (2025)
Keywords: computer vision
visual prompt
Primary area: transfer learning, meta learning, and lifelong learning
TL;DR: This paper systematically investigates current visual prompting methods and introduce a novel visual prompting approach for more efficient model adaptation

Abstract: Visual prompting has gained popularity as a method for adapting pre-trained models to specific tasks, particularly in the realm of parameter-efficient tuning. However, existing visual prompting techniques often pad the prompt parameters around the image, limiting the interaction between the visual prompts and the original image to a small set of patches while neglecting the inductive bias present in shared information across different patches. In this study, we conduct a thorough preliminary investigation to identify and address these limitations. We propose a novel visual prompt design, introducing **Lo**w-**R**ank matrix multiplication for **V**isual **P**rompting (LoR-VP), which enables shared and patch-specific information across rows and columns of image pixels. Extensive experiments across seven network architectures and four datasets demonstrate significant improvements in both performance and efficiency compared to state-of-the-art visual prompting methods, achieving up to $6\times$ faster training times, utilizing $18\times$ fewer visual prompt parameters, and delivering a 3.1% improvement in performance.

Published as a conference paper at ICLR 2025
LOR-VP: LOW-RANK VISUAL PROMPTING FOR EFFI- CIENT VISION MODEL ADAPTATION
Can Jin', Ying Li", Mingyu Zhao', Shiyu Zhao', Zhenting Wang', Xiaoxiao He²,
Ligong 3, Tong Che². Dimitris N. Metaxas
|Rtggers University, *Zhejiang University, RRed Hat Al Innovation, MIT-IBM Watson AI Lab, SNVIDIA Research
ABSTRACT
Visual prompting has gained popularity method for adapting pre-trained models to specific tasks, particularly in the realm of parameter-efficiem tuning. However, existing visual prompting techniques often pad the prompt parameters around the image, limiting the interaction between the visual prompts and the orig- inal image small set of patches while neglecting the inductive bias present shared information across different patches. In this study, we conduct thorough preliminary investigation to identify and address these limitations. We propose a novel visual prompt design, introducing Low-Rank matrix multiplication for Visual Prompting (LOR-VP), which enables shared and patch-specific informa- tion across rows and columns of image pixels. Extensive experiments across seven network architectures and four datasets demonstrate significant improvements in both performance and efficiency compared state-of-the-arr visual prompting methods, achieving up to 6x faster training times, utilizing 18x fewer visual prompt parameters, and delivering 3.1% improvement in performance. Code is available at wtlhtps::/ http:::///tttbbt htt.hhbb...mmm/mmmam33///////
INTRODUCTION
Many applications in computer vision (CV) and natural language processing (NLP) rely on adapting large-scale, pre-trained models to multiple downstream tasks (Liu et al., 202T; Dosovitskiy al.) 2020 Ridnik et al. 2021 Brown et al. 2020; Peng et al.) 2024). Recent advances in large language models (LLMs) have highlighted data-centric techniques such as in-context learning (Brown al.]
2020 Shin et al., 2020 Liu et al. (022) and prompting (Li & Liang 2021 Liu et al. 2023). These techniques well dell-nes prompts or input templates to significantly enhance the performance of LLMs across wide range of tasks. Inspired by these methods, visual prompting has gained sub- stantial attention as a means of adapting pre-trained vision models by modifying input pixels or output transformations (Bahng et al. 2022, Chen al.
2023 Tsao al. 2024).
Existing visual prompting methods, such as those proposed by CLIP-VP (Bahng et al. 2022). ILM- VP (Chen et al. 2023), and AutoVP (Tsao al. 2024), have demonstrated the capability to enhance the performance of pre-trained vision models across various downstream tasks. natural question arises: why does the addition padding of tunable parameters the original image pixels improves adaptation performance? A plausible explanation is that the introduced visual prompts (VPs) pro- vide task-specific information that not only alters the representation of the original images but also influences the attention distribution across image patches. This is particularly significant in pre- trained models such as Vision Transformers (ViTs) (Dosovitskiy et al., 2020), where VPs interact with patch tokens and guide the model's attention to differrnt parts of the image.
However, current VP designs primarily focus on adding or padding tunable parameters at the pe- riphery of the image (see Part I of Figure for existing VP designs), thus only allowing boundary patches be modified, while the central patches remain unchanged. These designs present two notable limitations: (1) The VP parameters are restricted to interacting with the original image in a limited set of patches, leaving substantial portion of the image unmodified. As a result, VPs
Correspondence Can Jin <can. meruutrrrree....

---

Published as a conference paper at ICLR 2025
only influence the model's interpretation of specific regions of the image, while other regions- potentially containing critical information-remall unaffected. (2) The VPs applied to each patch operate independently. disregarding the inductive biases present in the shared information and po- sitional encoding across different patches. For instance, one patch may represent part of an object, while adjacent patch represents another part of the same object. By ignoring these inter- onter-patch relationships, visual prompting may limit the model's ability to capture the global context of the image effectively.
To address these limitations, we first conduct a thorough investigation to understand the shortcom- ings of existing methods. Building on this analysis, we propose novel visual prompting method, termed LoR-VP, aimed at more efficient and effective adaptation of vision models. Our approach not only influences every patch of the image but also introduces inductive biases between the rows and columns of image patches. By leveraging LOR-VP, we achieve superior performance com- pared to state-of-the-art (SOTA) methods while significantly reducing the number of parameters required. In summary, our contributions are organized around the following three thrusts:
(Preliminary Study) We conduct an in-depth preliminary study to identify and illustrate the lim- itations present in current visual prompting methods and explore potential solutions to overcome these challenges.
(Novel Approach) To mitigate these limitations, we propose novel visual prompting technique, named LOR- VP,, which optimizes the visual prompts uniformly across all patches and introduces inductive biases between the rows and columns of the image patches.
* (Experiments) We perform extensive experiments across a wide range of large-scale models and datasets. The empirical results consistently demonstrate the significant improvements in both per- formance and efficiency achieved by LOR-VP, validating its practical effectiveness. For instance, LPR-VP surpasses the previous SOTA method, Au (Tsao et al. 20024). by average of 3.1% on seven network architectures and four datasets using 6 less training time.
RELATED WORKS
2.1 VISUAL PROMPTING
The concept of prompting originated in the field of NLP as a technique for adapting pre-trained
models to downstream tasks (Shin et al. 2020: Liu al. 2022 Li & Liang 2021 Liu et al. [023]
Han et al., 2023 2024 Uin et al.) 2023). This design philosophy was later extended to CV by Bahng et al. (2022). , who introduced tunable parameters directly into imput images create what is
known as Visual Prompt (VP). A typical VP framework consists of two primary modules: input
design and output transformation (Bahng al,, 2022 Tsai et al., 2020 Tsao et al. 2024 Cai al.) Various strategies have been proposed for constructing VPs. For instance, Baann et al. 2022 modify input images by adding frame of visual prompting parameters, whereas Chen et al.
2024
2023 incorporate the visual prompting parameters around resized images. Wu et al. (2022) explore elficient methods for generating visual prompts that enhance performance across different tasks, and Oh et al. (2023) develop visual prompts designed for adapting models to black-box, inaccessible models. Since the output logits of pre-traie pre-teained models remain in the source domain, an additional output transformation (e.g.. label mapping) is required to accurately predict the targets. A simple approach is to randomly map source labels (RLM) onto target labels. (T'ai et al.) (2020) propose a frequency-based label mapping (FLM) technique, which derives the mapping based on frequency statistics. Chen et al. (2023) further introduces iterative label mapping (ILM), which improves the performance of visual prompting. Yang et al. (2023) proposes semantics-baes based label mapping approach that aligns source and target classes based on semantic similarity. Additionally, [Taso et al. (2024) introduces full mapping (FM), utilizing an automated system to select the most appropriate label mapping (LM) method to optimize performance across diverse downstream tasks.
2.2 LOW-RANK STRUCTURES IN DEEP LEARNING
Low-rank structures are widely observed in machine learning, as many problems inherently ex- hibit low-rank properties (Li et al. 2016 Cai al. 2010 Li et al., 2018 Grasedyck et al. 2013).
2

---

Published as a conference paper at ICLR 2025
It has been found that for numerous deep learning tasks, especially those involving heavily over- parameterized neural networks, the resulting models tend to exhibit low-rank characteristics after training (Oymak et al.] 2019; Khodak et al. (02T). Some prior work has explicitly integrated low- rank constraints during the training process of neural networks (Sainath et al.) 2013] /Zhang et al.) 2014 /hho et al., (016). From theoretical perspective, neural networks have been shown to outper- form classical learning methods, including finite-width neural tangent kernels, when the underlying
concept class has low-rank structure (Allen-Zhu et al.) 2019 Li & Liang 2018 Ghorbani al.
2020 Allen-Zhu & Li 2019 2020). Additionally, |Allen -hu et al. (2200) highlight that low-rank adaptations can be beneficial adversarial training scenarios. The LoRÁ method, introduced by Hu et al. (2021), along with its variants (Zhang] 2023 Yeh 2023), is particularly noteworthy for not introducing additional inference burdens, thus improving the parameter efficiency of adapting large pre-trained models. These methods employ low-rank matrices to approximate weight updates during fine-tuning, enabling a seamless integration with pre-trained weights prior to inference.
PRELIMINARY STUDY
3.1 ANALYSIS OF PAD PROMPTING
Visual prompting is proposed to address the problem of adapting a pre-trained source model to downstream tasks without fine-tuning the network weights. Consider downstream target image dataset D = ((x,,1))..... v1..... (Xn; Un)} with typical color channels (usually 3) and pre-trained vision model with a resolution of L x (the default value of 224 for simplicity). Visual prompting modifies the images by adding tunable parameters to the image pixels. Among various methods, Pad Prompting Tsao al. 2024 Chen et al. 2023 Bahng et al., 2022) is the most preva- lent, which involves resizing the initial images to a specific size (Lypically smaller than L, such as 128), and then surrounding the resized image with a tunable parameter border size p such that + 2p L, resulting in prompted image of dimensions L x L. An illustration of Pad Prompting depicted in Part 1 of Figure [ The optimal values for & and p generally vary across different models and tasks. Vu, currently the SOTA in visual prompting, automates the selection of and p to enhance performance across various models and tasks (ssa et al. (2024).
1
Target Domain Data
Target
Target Domain
Target Domain Data
Input Scaling
Resize
Resizing Scale
1.0
x
1.5
24
Split
168
224 224
32 x 32
Split
p A
p-
Padding
32
Split into fixed size patches
x 7 patches, repeatedly
Visual Prompt
Visual Prompt
Visual Prompt
Prompt
AutoWP(Pad]
Patch-Pad
The value
is
Patch-Frre
Patch Same
Image Encoder
The value of is
Frozen
Tuned
The tunable
Figure 1: Illustration of various visual prompting methods applied target domain data: 1 Au- tPPa()::): Focuses optimizing the balance between image scaling and tunable parameter in- tegration to enhance model responsiveness. surrounding each image patch with tunable visual prompts. @ Patch-Free: Provides maximum adaptability by allowing independent tuning of visual prompts for each patch, catering to diverse feature requirements across the image. @ Patch-Same: Promotes consistency in model training by applying uniform visual prompts across all patches, ensuring coherent feature learning across input.
Patch-Pad: Aims to enhance localized learning by
Despite AutoVP's automated process for optimizing prompt size, Pad Prompting inherently only ad- justs the prompts in the peripheral patches of the resized images, leaving the central part unchanged.
3

---

Published as a conference paper at ICLR 2025
Furthermore, the parameters in different patches optimized independently, disregarding the in- ductive biases that arise from shared information and positional encoding across patches. We hy- pothesize that more effective visual prompting strategy would allow interaction with each patch while also considering the inductive biases among them. This hypothesis has led us to explore new VP designs:
Patch-Pad: As shown in Part of Figure 1. after resizing the image, we evenly split into different patches and then pad the tunable parameters around each patch to form resolution L x image. For instance, when using an ImageNet-21K (Deng et al. 2009) pre-trained ViT-B/32 (Dosovitskiy et al. (020) and fine-tuned on ImageNet- IK model, the resize image is split into 7 x 7 patches, and each patch is padded to a size of 32 x 32 using tunable parameters. The Patch-Pad method can influence each patch of the image but split the image into discontiguous parts and potentially devastate the information in the images.
Patch-Free: As shown in Part 3 of Figure D to avoid dropping information in the images, we directly resize the image to resolution L x L, then evenly add independent tunable parameters (initialized as to each patch of the resized image. The Patch-Free design can influence each patch of the image without splitting the resized image thus maintaining the information of the image, but this design updates the patch VPs independently and doesn't t consider shared information on different patches.
Patch-Same: As shown Part 4 of Figure I to enable shared prompting among each patch, we initialize patch of tunable parameters and repeatedly add it to all patches of the image. Patch-Same enables shared visual prompting among different patches but constrains the shared information as the same for all patches.
3.2 PERFORMANCE INVESTIGATION OF DIFFERENT VP DESIGNS
To investigate the performance of the aforementioned four VP designs, we conducted experiments using ImageNet-21K pre-trained ViT-B/32 and ViT-B/16 (both fine-tuned on ImageNet-IK K)) on CI- FAR10/100 (Krizhessky et al.) 2009). The performance benchmark, with all methods employing a fulll connected label (FM) (Tsao et al., 2024) for fair comparison. The results, depicted in Figure 2, indicate that Patch-Pad underperforms across all models and datasets, the most likely reason is it split the image patches thus might damage the image information. @ Patch-Free outperforms Patch-Pad, confirming that maintaining image continuity is beneficial. However, Patch-Free is less effective than Patch- prompting can enhance performance. @ Patch-Same outperforms Auto VP, underscoring the impor- tance of shared prompting information across patches. 1 The performance gap between Patch-Same and Pad Prompting shrinks in models that have smaller patch sizes, suggesting that when using the ViT-B/16 model, Patch-Same constrains more patches learn smaller visual prompt, which may bring too strong constraints to the visual prompts, and a better way is utilizing visual prompts that not only introduce inductive bias in different patches but also allow for pptch-specific visual prompting.
AutoVP (Pad Prompting) was used as
which suggests that shared visual
VIT-B/32,CIFARIO
VIT8/32,CIFAR100
VIT-B/16,CIFARD CIFARIO
VIT BTTB/16,CIFARIU
97,50
96
96.98
90
88.48
86
61
97.82
97.60
90
89,
88 84
86 65
95
Accuracy
65
Accuracy
95
Accuracy
88.84
80
90
80
75
30
38
69.9
90
70
85
85
60
60
100 Epoch
150
200
100 Epoch
150
200
50
100 Epoch
150
200
50
150 200 Epoch
100
AutoVP (Pad)
Patch-Pad
Patch-Free
Patch-Same
Our
Figure 2: Preliminary Investigation Results. Performance comparison of various VP designs. Our
VP method demonstrates competitive superior performance in several configurations. The final performance of each method is marked by * or e, with all results averaged over three runs.
4

---

Published as a conference paper at ICLR 2025
4 METHODOLOGY
Inspired by the observations in Section 3 we propose a novel visual prompt design that facilitates prompting across all patches while enabling both shared and patch-specific information. This ap- proach leverages low-rank matrix multiplication to efficiently manage visual prompts.
4.1 LOW-RANK VISUAL PROMPTING
In order to facilitate comprehensive pixel information and interaction across all patches, we resize the image to uniform size of L. x L. This resizing strategy is designed to minimize information loss, which is a common issue with the traditional Pad Prompting method that often resizes images to dimensions smaller than L. To enable the sharing of visual prompt information across different patches, we introduce two low-rank matrix parameters, B E Rxx lxxr and A E Rexxx where L. The product of these matrices, B A, serves the visual prompt.
This configuration allows the visual prompt to act as a linear combination of the row vectors in A and the column vectors in B, facilitating shared information across the rows and columns of the image. Additionally, this design supports patch-specific information, as the coefficients of each row and column are independently adjustable. Based on observations in Section 3 this approach to visual prompting is likely to yield superior performance. The visual prompt is directly added to the resized image, resulting in the prompted image being expressed as:
P(x) Resizez (x) BBA,, XED,
where Resizer(-) resizes the image x into a size of T, x L,, and matrices B and A are the initialized visual prompt parameters, we utilize zero initialization B and a random Gaussian initialization of A B A is zero at the beginning of training.
Utilizing a rank r 4 in B and A, we conduct experiments using the same configurations as Section 3 From the experimental results shown in Figure we can observe that our Low-Rank ma- trices multiplication Visual Prompting (LoR-VP) achieve the best performance among all designs, further validate our hypothesis in Section B
This method simplifies the visual prompting process compared to Pad Prompting, which involves complex resizing, padding, and mask manipulations. By employing low-rank matrices, reduce the number of tunable parameters from cL2 to crL, enhancing parameter efficiency significantly.
4.2 OUTPUT TRANSFORMATION
The output of the pre-trained model f on the prompted image P(x) remains in the source domain. To align these predictions with target labels in downstream tasks, we apply output transformation, denoted as M:
minimize S.M
We utilize Linear Probing (L.) as our output transformation method, which adjusts the output fea- tures of the classifier head to suit downstream classes. This method proves more efficient than ex- isting methods such iterative label mapping (ILM) (Chen et al.) 2023) and fully connected layer mapping (FM) (Tsa et al., 2024) on large models and datasets. For instance, when using ImageNet- 21K pre-trained Swin-B (Liu et al. 2021) and tuning ImageNet- IK, ILM costs much time and GPU storage to calculate and store the mapping sequences (a 21, 841 x 1, 000 matrix) and Au- VP also achieves inferior performance due to the ineffectiveness learn 21, 841 x 1, 000 full connected layer (see Figure for performance details).
5
EXPERIMENTS
To assess the effectiveness and efficiency of our novel visual prompting method, we adopt the most widely used evaluation protocol for VPs, i.e., evaluating models pre-trained on large datasets across
5

---

Published as a conference paper at ICLR 2025
Split into fixed size patthes
Original
Σ Bie Ans
VPu
224
Our Visual Prompt
Visual Prompt
Split into fixed size patches
Figure 3: Our VP Design. We resize the image resolution of L x L and initialize two low-rank matrices B and A as tunable parameters. B A serves as the visual prompt and is directly added the resized images. This design allows shared information in rows and columns and also allows patch-specific information in different patches.
various visual domains. Unlike previous works such as ILM-VV (Chen al. 2023) and Auto (Tsao et al.) 2024), which primarily utilize ImageNet-1K (Deng et al. 2009) pre-trained models and fine-tune on smaller downstream datasets, we extend our exploration to larger pre-training datasets, such as ImageNet-21K (Deng et al. 2009). well as larger downstream datasets, in- cluding ImageNet-IK, to examine the scalability of existing extensive empirical evaluations, focusing the following aspects: (1) Demonstrating the supe- rior performance and faster convergence of LOR-V -PP across different datasets and architectures; (2) Investigating the out-of-distribuutin robustness of LOR-VP. (3) Showcasing the efficiency of LoR- VP in terms of training epochs, runtime, and parameter usage, etc; (4) Performing ablation studies evaluate the effectiveness of our VP approach under various label mapping methods, the optimal rank configuration in LOR-VP, VP, and the contribution of different components within LoR- VP.
methods. Furthermore, we conduct
5.1 IMPLEMENTATION DETAILS
Datasets. For pre-training, we utilize the ImageNet- K dataset (Deng et al. 2009), which contains
1K classes and 1.3M images, the Imag dataset (Ridnik al. 2021), comprising 11K
classes and 12M images, and the ImageNet-21K dataset (Deng et al. (009). which includes 21K classes and 14M images. We evaluate the effectiveness and efficiency of LOR-VP across four down- stream datasets: ImageNet-IK, Tiny-ImageNet (Le & Yang 2013), and CIFAR- 10/100 (Krizhevsky al., 2009p. To assess the out-of-distribution robustness of LOR-VP, we conduct experiments on ImageNet-R (Hendrycks et al.) (021a)) ImageNet-Skett
(Wang et al., 2019), ImageNet-A (Hendrycks et al. (022b)), and ImageNet-V3 V2 (Recht et al. 2019). Additional details about the datasets are in Table
Networks. We employ six architectures for our experiments, all of which operate at resolution of 224 224. (1) ResNet-18 and ResNet-50 (He et al. 2016) pre-trained on ImageNet- and ViT- B/32 (Dosovitskiy et al. 2020) pre-trained with a classifier head of 1000 classes; (2) ResNet-50-P and ViT-B/16-P (Dosovitskiy et al. 2020), pre-trained 2IK with classifier heads for 11,221 classes; (3) Swin-B (Liu et al. 2021), pre- ioe-trained on ImageNet-21K,, with classifier head for 21,841 classes; (4) CLIP (Radford al. 2021), vision-language model that uses ViT-B/32 architecture as its vision encoder. The
mnageNet-21IK and fine-tuned on ImageNet-1K3 IK, each

---

Published as a conference paper at ICLR 2025
weights for these models are all publicly available through the official PyTorch Model Zod] or the Hugging Face Timm Library] Further details of the network architectures can be found in Table 7
Baselines. We select four representative SOTA methods as our baselines: (1) CLIP-VP (Bahng
et al. 2022), which extends prompt tuning to the computer vision domain by incorporating prompt parameters directly into input images using CLIP models; (2) ILM- VP (Chen et al., 2023). which explores the impact of frequency-basee
label mapping (FLM) in visual prompting and introduces iterative label mapping (ILM) for improved performance; (3) AutoVP (T'ao et al.) (024). a SOTA method in visual prompting that automates the selection of VP configurations, including prompt sizes and label mapping (LM) strategies- experiments use the optimal configuration provided by Vu (4) LP, which modifies the classifier head of the pre-trained model to adapt to downstream tasks, a commonly used technique in transfer learning, serving as a baseline akin to LoR- without the novel VPs introduced in our approach.
Training and Evaluation. The results for the baseline methods, including CLIP-VP, ILM-VP, and VP, are reproduced using the same configurations as described in their respective original papers. For LOR-VP. we resize all input images to 224 x 224 and use rank of in our VP design. As result, the two sets of parameters in LoR-VP have dimensions of 3 x 224 x 4 and 3 x 4 x 224, respectively, meaning that the total number of parameters in the visual prompts only 5K. The optimal hyperparameters for LOR-VP are determined through grid search. All experiments are conducted on NVIDIA Quadro RTX8000 GPUs with 48GB of memory. Additional implementation details for LoR-VP are provided in Table g
5.2 MAIN RESULTS
Performance of TetgeNet-IKK and CLIP Pre-trained Models. To demonstrate the effectiveness of LOR-VP on widely used ImageNet- pre-trained models, we evaluate its performance across several downstream datasets using ImageNet-21K pre-trained ViT-B/32 (fine-tuned on ImageNet- IK), ImageNet-IK pre-trained ResNet-18 and ResNet-50, as well as CLIP models. As shown in Figure 4 we can observe that: O LOR-V VP consistently outperforms all baselines across all network and dataset combinations, achieving an average improvement of 3.2% and 2.1% over Auto VP and I.,, respectively. LOR- converges significantly faster than the baselines, reaching optimal performance with 5x fewer training epochs than PP and 10x fewer than ILM-VP.
CLIP, Imp.Teny
CLIP,CIFARD CIFAR10
VIT-B/32, Tinggg..Tnyy-mmaeNNet
VIT-B/32,CIFARI CIFAR100
75
71,58
85
81.59
85,85
83
90
88
86,
Accuracy
67.44
26
Accuracy
79.43
75.00 04
Accuracy
82.43
Accuracy
70
85.96
65
75
65.66
50
50
100
150
200
0
50
100
150
200
50
100
150
200
0
50
100
150
200
Epoch
Epoch
Epoch
Epoch
ResNet-5D, Tnyy Imny-lma
DesNet-SO,.IIART
ResNet- 18, Iinn
ResNet- ResNet-18,CIFAR 18,CIAAR0
77.55
77 04
92.35
91.41
70
68.28
65
90
88.64
67 55
Accuracy
70
76.52
Accuracy
90
91.35
Accuracy
50
59.70
Accuracy
60
67.11
50
37.74
80
30
70
66 39
14:13
30
70
10
100
150
200
0
50
100
150
200
100
150
200
0
50
100
150 200
Epoch
Epoch
Epach
Epach
AutoVP
ILM-VP
CLIP-VP
LP
LoR-VP
Figure 4: Performance of ImageNet-1K and CLIP Pre-trained Models on Downstream Datasets. Overview of the performance of LOR-VP compared four baseline methods. The fi- nal performance of each method is indicated by * and all results are averaged over three runs. LoR- VP consistently outperforms all baselines across various models and datasets.
7

---

Published as a conference paper at ICLR 2025
Performance of ImageNet-21K Pre-trained Models. To further evaluate the effectiveness of LOR- VP and existing visual prompting methods on larger models and datasets with a greater num- ber of classifier classes, conduct experiments using ImageNet-21I--B pre-trained ResNet-50-P and ViT-B/16-P, as well as ImageNet-21K pre-trained Swin-B models, tuning them on ImageNet- 1K and Imay--m These models have significantly more classifier output features compared to ImageNet-IK pre-trained models, providing additional evidence of the effectiveness of LoR- VP and other VP methods on large-scale models and datasets. For the ImageNet-IK experiments, we focus on the strongest baselines, such as Au and LP, running them for 30 epochs in line with the implementation [iu al. (022), due resource constraints. We found challenging to run ILM-VP on our GPUs, the ILM process is computationally expensive in terms of both training time and GPU memory. The results of the experiments are presented Figure where we observe the following: t LOR-VP V consistently achieves the best performance across large models and datasets, outperforming all baselines. @ The performance gap between LoR- VP and Auto VP increases to 5.06 ImageNet-Il IK. A likely explanation is that the full mapping (FM) method used in Auto VP is less effective in this scenario, as it struggles to efficiently train a fully connected layer with 21, 841 input features and 1, 000 output features.
VIT-B/16-PP.
6...///6..Tiy-lBi. Impy-Imagggn Net
ResNet50 P...eet--...Tinlmmme P.inylmmgeeee
Swin-B, Dovin-,.Tiny-Imaen Imageee
Swin ImageNet
84,
75
78.42
77
86.52
56.54
84,04 04
80
Accuracy
81.42
Accuracy
75.67
Accuracy
84.81
Accuracy
60
80
70
78,98
40
25
40
27.81
56
75
20
50
100
150
Epoch
Epoch
Epach
Epoch
AutoVP
ILM-VP LP
LoR-VP
Figure 5: Performance of ImageNet-21K Pre-trained Models on ImageNet-1K and Tiny- ImageNet. Performance comparison of LoR-VP and four baseline methods. The models pre- trained on either ImageNet-21K: or ImageNet-21 and then tuned on the respective downstream datasets. The final performance results are denoted by * All results are averaged over three runs. LOR-VP consistently outperforms all baselines across different models and datasets.
5.3 ROBUSTNESS OF LOR-VP
To investigate the out-of-distribution robustness of LOR-VE VP and explore its potential for enhancing real-world applications, we conduct experiments using ImageNet-22 pre-trained Swin-B. We apply LOR-VP on ImageNet-IK and then evaluate the performance of the resulting model and visual prompts on four out-of-distribution datasets. The performance of LOR-VP and the two strongest baselines are presented in Table [ of-distribution robustness and generalization performance across all baselines, achieving an average improvement of 10.6 over AutoVP across the four datasets. These results further highlight the superior -idt-----istttttion robustness of LOR-VE VP, confirming its advantage over SOTA prompting methods in terms of generalization.
We find that LOR-VP consistent demonstrates the best out-
Table I: Out-of-Distribution Generalization Performance. Evaluation of the -if-o---iitriuttion generalization performance using the ImageNet-21K pre-trained Swin-B, with visual prompting ap- plied on ImageNet-1K K, and tested across four out-of of-diitiibtttionn datasets.
Source
Target
Method
ImageNet-1K
ImageNet-R
ImageNet-Sketch
ImageNet-A
ImageNet-V2
Au [ICLR24]
78.98
38.
28.89
17.91
67.38
83.52
51.48
40.07
27.54
71.86
LPR-VP
84.04
52.27
41.13
27.89
72.38
8

---

Published as a conference paper at ICLR 2025
5.4 EFFIEIENCY OF LOR-VP
As shown in Figures and 5 LOR-VP VP achieves superior performance compared to SOTA meth- ods with fewer training epochs. To further examine the efficiency of LoR-VPin contrast to the baselines, we assess its performance using several criteria: training epochs, training time, tunable parameters (including visual prompt parameters), GPU memory usage during training, and infer- ence latency. Evaluations are conducted using an ImageNet-2IK-P pre-trained ViT-B/16-P with vi- sual prompting applied on Tiny-ImageNet, and an ImageNet- 1K pre-trained ResNet-18 with visual prompting applied on CIFAR-10. 10. The results are presented in T we can observe that: ! LoR- VP converges the fastest among all methods, requiring 5x fewer epochs and 6x less training time compared to VP, and 10x fewer epochs and 15x x less training time compared to ILM-VP. ILM- VP, in particular, converges the slowest and incurs the highest time cost, as it requires an additional epoch for every training epoch to compute the LM sequences. @ LoR- VP is highly parameter- efficient. For small models and datasets, such as ResNet-18 and CIFAR- 10, LOR- VP only requires 10K parameters achieve optimal performance, which is 15x and 11 fewer than ILM and VutoVP, respectively. Notably, LOR-V VP requires just 5K visual prompt parameters, which is 18x and 30x fewer than AutoVP and ILM-VP, on average. ① GPU usage and inference speed for LOR- VP are comparable to Au whereas ILM-VP consumes the most GPU memory on larger models due to the additional computation and storage required for LM sequences. 2 LoR- VP achieves the best performance with the fewest visual prompt parameters and the shortest training time, mak- ing an ideal choice for adapting pre-trained vision models to downstream tasks, particularly for resource-constrainee environments such as mobile devices.
Table 2: Training and Inference Efficiency. Comparison of the training and inference efficiency of LoR-VP, VutoVP, and ILM-VP, evaluated using 1mageNet-211-P pre-trained ViT-B/16-P on ImageNet and ImageNet- IK pre-trained ResNet- 18 on CIFAR-10.
Network
Method
Epochs
Params
Tunable Params
CPU Usage
Latency
Ascuracy
ILM-VPICV
200
5.76h
L47K
147K
4.51GB
4.61ms
66.39
R8 CIFARIO
AutoPPIICI R24)
100
2.61h
101K
111K
4.5GGB
4.59ms
87.55
LoR
20
5K
10K
4,49GB
4.53ms
88.
ILM- VPCCPP23)
200
25,55h
L47K
147K
17.24GB
14.55ms
26.17
VIT-B/16-P
Tiny-InageNer
AutoVPIiCI R24]
100
8.08h
74K
2
13,39GB
14.40ms
81.42
LOR-VP
159K
13.34GB
14.29ms
84.40
5.5
ALBATTON STUDIES
How does Output Transformation Impact LOR-VP's Performance? To further explore how different output transformations affect the performance of LOR-VP, we conduct experiments by combining LOR-VP with FLM, ILM, and FM, referred to as LOR-VPw. FLM, LOR- VP w. ILM, and LOR- VPw. w. FM, respectively. These experiments are performed using ImageNet-21K pre-trained Swin-B, ImageNet-21K-P pre-trat pre-trained ViT-B/I6-P, ImageNet-21K pre-trained ViT-B/32 (fine-tuned on ImageNet- and ImageNet-IK pre-trained ResNet-18 on CIFAR-100 and Tiny- ImageNet. The results are presented in Table where we observe the following: ① LOR-VP with LP as the output transformation achieves the overall best performance across all methods, net- works, and datasets. @ Even when using the same output transformations as ILM-VP and Auto LOR-VP consistently outperforms these methods, further demonstrating the superiority of our vi- sual prompt design.
What is the optimal rank in LOR-VP? To provide deeper insights into the optimal rank selection LOR-VP we conduct experiments with various configurations: LPR LoR- VP combined with ILM as the output transformation named as LOR-VP w. ILM, and LoR- VP combined with FM as the output transformation named as LoR- VP w. FM. These experi- ments are performed using the ImageNet-21K- K-P pre-trained ViT-B/16-P on Tiny-ImageNe and the ImageNet-21K pre-trained VIT-B/32 (fine-tuned on ImageNet- on CIFAR-100. The results are presented in Figure 6 from which we can derive the following observations: ! For output transfor- mations such as LP and FM, the optimal rank is 4; increasing the rank beyond 4 does not yield any further performance improvements. @ When LoR- VP is combined with ILM, the optimal rank is around 16. A plausible explanation is that ILM lacks tunable parameters, so a higher rank is needed to enhance the expressive power of the visual prompts and achieve optimal performance.

---

Published as a conference paper at ICLR 2025
Table The Impact of Output Transtormation. The performance comparison of utilizing FLM, ILM, and FM as the output transformation of LP and the baselines. LoR- VP achieves the overall best performance among all output transformation methods, networks, and datasets.
Network
Dataset
Method
Outpur Transformation
Swin-B
ViT-B/16-P
VIT-B/32
ResNet-11
ILM ILM-VPICVPR23) AutoVPIICLR24]
ILM
56.28
26.17
32.58
14.13
FM
84.81
81.42
82.43
$9.68
LP
LP
86.54
82.75
83.95
65.17
Tiny-ImageNet
L. FLMM LOR- VP w. ILM LOR-VP FM
FLMM
82.15
41.76
82.89
57.45
ILM
84.85
43.50
84.86
62.20
FM
85.59
83.15
86.03
65.63
LOR-VP
LP
88.28
84.40
85.85
68.28
U.M-VICCPPRA AUIVP(|ICR24)
ILM
65.78
41.49
40.10
25.36
FM
86.83
88.58
85.96
63.77
LP
LP
87.37
88.90
86.21
67.06
CIFAR100
LOR-VE VP W. FLM
FLM
74.08
46,35
72.81
34.58
LOR-VE w. ILM
ILM
77.22
48.53
78.23
39.07
LOR-VE VP w. FM
FM
86.25
89.10
88.48
68. 64
LOR-VP VP
LP
90.42
89.69
88.65
69.88
Ablation of Components in LOR-VP. We perform ablation studies on the two key components of LOR-VP: the low-rank VP design and the linear probing output transformation. For experi- ments without label mapping, we apply FLM prior to training and keep this mapping sequence fixed during visual prompt training to ensure valid results. We conduct these experiments using the 2 pre-trained ViT-B/16-P, ImageNet-2IK pre-trained Swin-B, and ImageNet-IK pre-trained ResNet-18 on Tiny-ImageNet. The results, shown Table 5 reveal the following: O LoR- VV- achieves the highest performance when both the low-rank VP and the output transfor- mation are employed, demonstrating the effectiveness of these components LOR-VP.. @ Our vP design improves model performance, regardless of whether fixed mapping sequence linear probing is used.
NTTB/32.CIFAR100
VIT-8/16-P, (TTB//16P,Tiny--maaaeee ImageNet
Table 5: Ablation of Components in LoR- VP. The results of using ImageNet-21K-P pre- trained ViT-B/16-P, ImageNet-21 pre-trained Swin-B, and ImageNet-IK pre-train ResNet- 18 on Tiny- Imageeet..
84
Accuracy
Accuracy
83
82
80
44
Components
Network
70
VP Output Trans. ViT-B/16-P Swin-B ResNet- 18
logz (Rank)
logz (Rank)
X
36.09
80.61
40.79
LaR-VP
w.
LoR-VP
/
x
41.76
82.15
57.45
X
82.75
86.54
65.17
Figure The Impact of Rank LOR-VE
/
84.40
88.28
68.28
6 CONCLUSION
Visual prompting has emerged powerful technique for adapting pre-trained models to specific tasks through parameter-efficient tuning. Traditional methods, however, often restrict the interaction between visual prompts and the original image to a limited number patches, overlooking the potential benefits of shared information across different patches. Addressing these shortcomings. study introduces a novel approach, termed (LoR-Vo VP), which facilitates both shared and patch-specific information dissemination throughout the image. Extensive experiments over seven networks and eight datasets consistently demonstrate the effectiveness and efficiency of our method.
Matrix Multiplication for Visual Prompting
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENTS
This research partially funded by research grants to Metaxas from NSF: 2310966, 2235405, 2212301, 2003874, 1951890, AFOSR 23RT0630, and NIH 2R01HL.
REPRODUCIBILITY STATEMENT
The authors have made an extensive effort to ensure the reproducibility of the results presented in the paper. First, the details of the experimental settings are provided Section5.I in the Appendix A This paper investigates eight datasets, and the details about each dataset are described Table The evaluation metrics are also clearly introduced in Section 5.1 Second, the baseline methods implementation particulars elucidated in Section [.. Simultaneously, the implementation details of our method, LoR-VP, VV,, are included in Section 5.1 and Appendix A Third, the codes are included in the supplementary material for further reference.

---

Published as a conference paper at ICLR 2025
REFERENCES
Zeyuan Allen-Zhu and Yuanzhi Li. Can SGD learn recurrent neural networks with provable gener- alization? In Advances in Neural Information Processing Systems (NeurlPS), pp. 10310-10320, 2019.
Zeyuan Allen-Zhu and Yuanzhi Li. On the backward stability of SGD and its use for asymptotic model selection in neural networks. Journal of Machine Learning Research (JMLR), 21(190): 1-50, 2020.
Zeyuan Allen-Zhu, Yuanzhi Li, Zhao Song, and Yingyu Wang. On the convergence rate of training recurrent neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pp- 10031-10041, 2019.
Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. Backward feature correction: How deep learning performs deep learning. In Proceedings of the 33rd International Conference on Neural Informa- tion Processing Systems, pp. 13540-13550, 2020.
Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale scale models. arXiv preprint arXiv:2203.12274 2022.
Alexey Bochkovskiy, Chien- Yao Wang, and Hong- Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004. 10934, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle. M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc., 2020. URL httpps::/ppocceee...oss wnw.:eeeedddd.oo...nee........e. wttpsuripsscccpppepeo cc/paper filess/ppper/2220// $1244570066f433111110aa222222222
Chengyi Cai, Zesheng Ye, Lei Feng. Jianzhong Qi, and Feng Liu. Samplc-specific masks for visual reprogramming-baae based prompting. In Forty-first International Conference on Machine Learning, 2024.
Jian-Feng Cai et al. Singular value thresholding algorithm for matrix completion. SIAM Journal on Optimization, 2010.
Aochuan Chen, Yuguang Yao, Pin- Yu Chen, Yihua Zhang, and Sijia Liu. Understanding and improv- ing visual prompting: A label-mapping perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19133-19143, 2023.
Liang- Ching Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder- decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp. 801-818, 2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 218-248225 255. leee, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020.
Amirata Ghorbani, Song Mei, Theodor Misiakiewiez, and Andrea Montanari. Neural networks are more expressive than kernel methods: A representational perspective. In International Conference Learning Representations (ICIR), 2020.
12

---

Published as a conference paper at ICLR 2025
Lars Grasedyck al. Low-rank tensor approximation techniques. SIAM Journal Matrix Analysis and Applications, 2013.
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 7323-7334, 2023.
Ligong Han, Song Wen, Qi Chen, Zhixing Zhang. Kunpeng Song, Mengwei Ren, Ruijiang Gao, Anastasis Stathopoulos, Xiaoxiao He, Yuxiao Chen, et al. Proxedit: Improving tuning-free real image editing with proximal guidance. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 4291-4301, 2024.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Kaiming He. Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, al. The many faces of robustness: A criti- cal analysis of out-of-distribution generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 134008349, 2021a.
Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- tion, pp. 15262-15271, 2021b.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 09685, 2021.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision (ECCV), 2022. URLhttpss: /ttpss://arxivv org/ /ttp//bb/2..11111133
Can Jin, Tianjin Huang, Yihua Zhang, Mykola Pechenizkiy, Sijia Liu, Shiwei Liu, and Tianlong Chen. Visual prompting upgrades neural network sparsification: A data-model perspective. arXiv preprint arXiv:2312.01397, 2023.
Mikhail Khodak, David Macko, and Christopher De Sa. Initialization and regularization of factor- ized neural layers. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. 2015.
Xiang Li et al. Low-rank adaptation of large neural networks. International Conference on Learning Representations (ICLR), 2016.
Xiang Li et al. Low-rank matrix recovery. Foundations and Trends@ in Machine Learning, 2018.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4582-4597, 2021.
Yuanzhi Li and Yingyu Liang. Learning overparameterized neural networks via stochastic gradient descent on structured data. In Advances in Neural Information Processing Systems (NeurIPS), pp. 8157-8166. 2018.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot efffmeter-efficient fnnettning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950-1965 2022.
13

---

Published as a conference paper at ICLR 2025
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. ACM Computing Surveys, 55(9):1-35, 2023.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 10012-10022, 2021.
Changdae Oh, Hyeji Hwang, Hee-young Lee, YongTaek Lim, Geunyoung Jung, Jiyoung Jung, Hosik Choi, and Kyungwoo Song. Blackvip: Black-box visual prompting for robust transfer learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 24224 24235, 24224-24235, June 2023. URL ht httpt: httpt::::/ppppcccese.ttt tweccf.e.......... com/
alac -ox Vic__iiiii Visual Prompting.
Sewnnnsfee__errinng CVPR 2a022__ppeer______ html
Samet Oymak, Zalan Fabian, Mingchen Li, and Mahdi Soltanolkotabi. Generalization guaran- tees for neural networks via harnessing the low-rank structure of the jacobian. arXiv preprint 1xXiv:990600392,, 2019.
Hongwu Peng, Xi Xie, Kaustubh Shivdikar, Md Amit Hasan, Jiahui Zhao, Shaoyi Huang, Omer Khan, David Kaeli, and Caiwen Ding. Maxk-gun: gnn: Extremely fast gpu kernel design for acceler- ating graph neural networks training. In Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2, pp- 683-698, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8768. PMLR, 2021.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International conference on machine learning. pp. 2019.
PMLR,
Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-216 pretraining for the masses. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021.
Tara N Sainath, Brian Kingsbury, George Saon, Hagen Soltau, Abdel-rahman Mohamed, George E Dahl, and Bhuvana Ramabhadran. Low-rank matrix factorization for deep neural network training with sighenimmnsional output targets. In IEEE International Conference on Acoustics, Speech and Signal Processing, pp. 6655-6655 IEEE, 2013.
Taylor Shin, Yasaman Razeghi, Robert L.. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.). Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 4222-4235, Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. http:///0022000.2mmmmiiiee URL htap:::://aalaathologg.s org/ 2020 cannl-mmaii 346
Yun- Yun Tsai, Pin- Yu Chen, and Tsung- Yi Ho. Transfer learning without knowing: Reprogramming black-box machine learning models with scarce data and limited resources. In International ngs.
Conference on Machine Learning (ICML). PMLR, 2020. URL
ml mlr.press wwt.vw19//tttete w9/tssai20000 htmll
Hsi-Ai Tsao, Lei Hsiung, Pin-Yu Chen, Sijia Liu, and Tsung-Y Yi Ho. AutoVP: An Automated Visual Prompting Framework and Benchmark. In The Twelfth International Conference on Learning Representations, 2024.
Haohan Wang, Songwei Ge, Zachary Lipton, and Eric P Xing. Learning robust global representa- tions by penalizing local predictive power. Advances in Neural Information Processing Systems, 32, 2019.
14

---

Published as a conference paper at ICLR 2025
Lirong Wu, Cheng Tan, Zicheng Liu, Zhangyang Gao, Haitao Lin, and Stan Z. Li. Learning aug- ment graph structure for both homophily and heterophily graphs. In Proceedings of the European Conference on Machine Learning (ECML), 2022. URL https://link springer com/
article/10. 1007 s 0994
90-9
Hao Yang, Junyang Lin, An Yang. Peng Wang, and Chang Zhou. Prompt tuning for unified multimodal pretrained models. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 402-416, Toronto, Canada, July 2023. Association for Computational Linguis- peattps:/aaclatthheen org./000.../0....
tics. doi: 10. 1153//1112222333... 2.n../222..nniiigg-al22... findings-acl. .27
et al. Ych. Lora: A unified low-rank adaptation framework for stable diffusion. arXiv preprint arXiv:2409.1499 14983, 2023.
et al. Zhang. Svd-Ilm: Truncation-aware singular value decomposition for large language model compression. arXiv preprini arXiv:2403.07378. 2023.
Zhifei Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi-task learning. In European conference on computer vision, pp. 94-194--88 Springer, 2014.
Menglong Zhao, Wanli Ouyang. Xiaogang Li, and Xiaowei Wang. Energy-efficiend image clas- sification on low-power iot devices. IEEE Transactions on Circuits and Systems for Video Technology, pp. 205-215. IEEE, 2016.
15

---

Published as a conference paper at ICLR 2025
A IMPLEMENTATION DETAILS
Table Dataset Information.
Dataset
Original Resolution
# Training Set Images
# #Tsst Set Images
# Classes
ImageNet-211 Deng et 2009
Varies
14M
21,843
ImageNet-21K-4 K-P (Kidnik al. 2222
224 x 224
12M 1.3M
0.6M
11,221
ImageNet-1K KK Deng et al. 2009
Varies
50K
1,000
ImageNet-R [Hendrycks ct al.
202
Varics
30K
200
ImageNet-Skerra Sesiicc(aaanpe (Wang el al.
2019
Varies
50K
1,000
ImageNet-A (Hendrvcks ct al 2021a
Varies
7.5K
1,000
ImageNer-V2 ( (Kccht al. 2019
Varics
10K
1,000
SinyIImaeNNetLCC Yang
2015
64 x 64
100K
10K
200
CIFAR (Krizhevsky et 2009
32 x 32
50K
10K
100
CIFAR10 10 Krizhevsky al
2009
32 x 32
50K
10K
10
Table 7: Network Information.
Network
Clossifier Hoal Input
Duijui
Re
2016
ImngeNet-1K
224 224 224
1,000
50
ImageNet-Ll
2,048
1,000
-22
2,048
ImageNaa221K-P
-24
8/16
21K, EnsgeNes-IK
224
768
1,800
8/32
21K, EmageNec-11
224
768
Swis.
ImageNet
10HM
224
1,024
21,841
WchlmageTest
Table 8: Implementation Details.
Network
Pre-trained Daia
Downstream Daia
Resolution
Opsimizer
1R
Label Mapping
LoR- VP Rank
Epochss
Buich Sizee
ResNer-18
ImsgeNes-1K
CIFARIO0
224
SGDD
ResNa-50
CIFARI00
SGD
Prohing
ImageNes:216 ImageNet-IK
CIFARIOO
224
Probung
ResNer-SO.P
InapeNas-21K.
Tiny-ImageNer
224
SCDD
Prohing
256
VIT-B/IGP
InageNas-21KC
Tiny-ImageNer
224 224
SGD
002
Linea: Probing
20
256
Swin-B
ImagoNet:2119
Tiny-ImageNNt
221 221
SGD
0.02
Linear Prohing
20
256
CLIP
Weblmage Text
Tiny ImageNet
224 224
SGD
40
Linear Probing
20
256
Swin-B
ImegeNet-21K
ImsageNer-11 KK
234 224
SGD
0.1
Linesr Prohing
10
256
B ADDITIONAL INVESTIGATION
Visual Prompting Object Detection and Semantic Segmentation. In this paper, we primarily focus image classification tasks, following previous works such as AutoVP and ILM-VP. To further explore the applicability
LOR-VPto object detection and semantic segmentation tasks, conduct experiments using YOLOv4 (Bochkovskiy al. 2020) for detection and DeepLabv3+ (Chen et al.] 2018) for segmentation. Both models use ImageNet-1K pre-trained ResNet-50 as the backbone. Hyperparameters such as the number of epochs and the rank in LOR-VP are kept consistent with those used in classification tasks.
For object detection, train the Pascal VOC 2012 and 2007 training sets and evaluate the Pascal VOC 2007 test set, following the setup in He et al. (020)) The bounding box head is modified for output transformation, and learning rate of 0.0001 applied. For semantic segmentation, we train on the Pascal VOC 2012 training set and evaluate on its validation set, with the DeepLabv3+ head adapted for downstream segmentation and a learning rate of 0.01. The experimental results for detection are presented in Table 9 while the segmentation results are shown in Table TO Our method, LoR-VP, outperforms AutoVP by nearly 4% in AP30 on VOC 2007 detection and by 1.1% on VOC 2012 segmentation, demonstrating the effectiveness of LoR-V VP for object detection and semantic segmentation tasks.
Additional Investigation on Diverse Downstream Tasks. To assess the performance of LoR- VP across broader range of classification tasks, including those involving natural and artificial
16

---

Published as a conference paper at ICLR 2025
Table 9: Performance (AP, AP50. and APrs) for object detection using YOLOv4 with ImageNet- IK pre-trained ResNet-50 backbone, evaluated on the Pascal VOC 2007 test set.
Table 10: Performance (mIOU) for seman- tic segmentation using DeepLabv3+ with an ImageNet- IK pre-trained ResNet-50 backbone, evaluated Pascal VOC 2012 validation set.
Method
AP
AP
AP75
Method
mIOU
LP
42.87
75,25
47.74
LP
67.82
Au
41.72
73.07
44.85
Au
67.42
LOR-VP
43.21
77.02
48.07
LOR-VP
68.55
objects, scenes, and textures, we conduct experiments on ten downstream datasets. These experi- ments use ViT-B/32 pre-trained on ImageNet-21K and fine-tuned on ImageNet-IK, following the methodologies of Au and ILM-VP, to further evaluate the generalization and robustness of our approach. Although LoRR- VP primarily focuses on pixel-level visual prompt designs, we extend our comparison include VPT-DEEP, as described by Jia et al. (2022). which modifies the transformer layers. This allows for more comprehensive evaluation against additional baselines. The exper- imental results, presented in Table 11 show that LoR- VP achieves superior average performance across the ten datasets compared to VPT and Vu Specifically, LoR- VP improves performance by 1.4% over Au and 1.3% over VPT, further demonstrating its effectiveness in diverse scenar- ios and against wider range of baselines.
Table 11: Comparison of accuracy between LoR- VP and four baseline methods using ViT-B/32 pre-trained on ImageNet-21K and fine-tuned on ImageNet-16 K across ten datasets.
Tla-- -mageeett
EuroSAT
OxfoedPets
Flowerst02
CIFARIO
Average
83.95
95.67
82.18
97.
83.21
$7.05
[ECCV22]
95.90
72.11
51.48
ILM-VP (CVPR23)
32.58
88.12
78.92
48.24
42.65
64.27
85.27
40.10
80.81
67.88
62.88
AutoVP [ICLX24]
82.43
96.25
92.12
82.86
70.81
98.42
95.45
85.96
85.24
86.
87.59
LOR-VP
85.85
96.25
92.18
83.51
72.49
98.58
97.51
88.65
$6.31
88.07
$8.94
Table Additional comparison of LoR- and Au using LP and FM as output transforma- tions, respectively.
Network
Dataset
Method
Output Transformation
Swin-B
VIT-B/16-P
VIT-B/32
ResNet-11
LP
86.54
82.75
83.95
65.17
AutoVPTICLRO R224]
FM
84.81
81.42
82.43
59.68
AutoVP LPUCCR224]
LP
86.45
82.92
83.31
65.58
ImageNet
LoR-VPw. FM LOR-VP
FM
85.59
83.15
86.03
65.63
LP
88.28
84.40
85.85
68.28
LP Auto MUOVPICLRON R24] AutoVP w. LPHCCR224)
LP
87.37
88.90
86.21
67.06
FM
86.83
88.58
85,96
63.77
LP
88.70
89.34
87.00
68.10
CIFAR100
LOR-VPw w. FM
FM
86.25
89.10
88.48
68.64
LOR-VP
LP
90.42
89.69
88.65
69.88
Additional Investigation Output Transformation. Table presents the impact of output trans- formations on LP, demonstrating that LOR-VP outperforms baseline methods, such as Au- toVP and ILM-VP, when using the same output transformations. These results validate the effec- tiveness of our visual prompt designs. To further examine the influence of different output trans- formations and reinforce the superiority of our approach, we conduct additional ablation studies comparing LoR- and Au using LP and FM as output transformations. The experiments performed on the same architectures and datasets as those in Table 3 The results, shown in Table 12 indicate that LoR- RP consistently outperforms Auto VP with I.P the output transformation
17

---

Published as a conference paper at ICLR 2025
across all models and datasets. Interestingly, AutoVP with LP achieves higher performance than LP alone on CIFAR100 but performs comparably on Tiny-ImageNet. This variation may stem from the scaling factors employed in AutoVP, which likely affect visual prompting performance differ- size ently across datasets. Notably, LoR- VP adopts a fixed visual prompt size of 224 224, simplifying its design by avoiding the need to account for scaling size, further underscoring its simplicity and adaptability.
18


Reviews:
[{"review_id": "repE7UADBT", "reviewer": "Reviewer_NrTN", "summary": "The paper addresses the task of visual prompting for adapting pre-trained models to specific downstream tasks. The paper investigates the limitations of existing visual prompting techniques, which often based on padding. The paper also proposes a new visual prompting technique based on low-rank matrix multiplication.", "questions": "- Why are alternative approaches of sharing information across patches not explored? \n- The paper uses linear probing to transform the labels from the source to the target domain. This is a very simple model and it is not clear why this is more appropriate than other transform models.\n- If I understand correctly, all downstream tasks are image classification tasks. Would the approach be able to deal other downstream tasks, e.g. object detection or segmentation?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper identifies the limitations of existing visual prompting techniques, which often restrict interaction between visual prompts and the original image to a small set of patches.\n- A novel visual prompt design based on low-rank matrix multiplication is proposed. This design allows for shared and patch-specific information across rows and columns of image pixels.\n- The results are convincing and demonstrate performance and efficiency improvements. The authors include extensive experiments across seven network architectures and several datasets showing a performance improvement compared to state-of-the-art methods.\n- The paper is well written and clear.", "weaknesses": "- A low-rank matrix multiplication is just one way of sharing information across patches. I am surprised that other approaches have not been tested.\n- The conclusions in the paper are very superficial and do not offer a deeper insight into the experimental results and the strengths and weaknesses of the proposed approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9ylYcb3hbP", "reviewer": "Reviewer_pQoJ", "summary": "This paper proposes the incorporation of low-rank matrix multiplication in visual prompting, resulting in improved performance compared to existing visual prompting methods, as well as enhanced training speed.", "questions": "Please see weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. the paper is very well-written and easy to follow. The background part is very clear and detailed.\n2. the method is simple yet effect. And the author gives good preliminary analysis that motivates the method, which makes a lot of sense to me.", "weaknesses": "1. Regarding the pad-based method, the authors state, \"The VP parameters are restricted to interacting with the original image in a limited set of patches, leaving a substantial portion of the image unmodified.\" I find this assertion questionable. If the backbone is a ViT, the padded tokens will interact with the inner tokens through self-attention, potentially affecting the entire image.\n2. The authors do not include a comparison or discussion of visual prompt tuning [1], which is a more prevalent method than those cited in the paper.\n3. The dataset utilized in the out-of-distribution generation experiments is insufficient to demonstrate the robustness of the method. It employs several variants of ImageNet, which exhibit minimal domain gaps, and all tasks focus on general object recognition. I recommend using a benchmark similar to AutoVP [2], which includes fine-grained classification and domain-specific tasks for a more comprehensive evaluation.\n4. The comparison between AutoVP and LP appears unusual, as LP seems to outperform AutoVP in most cases. This contradicts the conclusions drawn in the AutoVP paper. Additionally, the performance gap between LoR-VP and LP is minimal. What would be the outcome if AutoVP's output transformation were replaced with LP?\n5. As a general PEFT method, the evaluation is limitated to image classification, without extension to other tasks such as segmentation, detection, or caption.\n\n[1] Jia, Menglin, et al. \"Visual prompt tuning.\" European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022.\n\n[2] Tsao, Hsi-Ai, et al. \"Autovp: An automated visual prompting framework and benchmark.\" arXiv preprint arXiv:2310.08381 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aBNkVVkvah", "reviewer": "Reviewer_PGL1", "summary": "This paper introduces Low-Rank Visual Prompting (LoR-VP), which uses low-rank matrix multiplication to generate visual prompt, enabling more efficient information sharing across image patches, taking the inductive biases among them into consideration. The authors conducted a preliminary study comparing current SOTA (AutoVP) with three new VP designs, proving that VP should combine the benefits of both patch-specific and shared visual prompting. Tested on several network architectures and datasets, the proposed approach reduces the number of tunable parameters by up to 18× and achieves up to 6× faster training times while improving performance by an average of 3.1%.", "questions": "Impact of Image Scaling vs. Patch-Specific Information: Can you provide additional experiments or controlled studies to isolate the impact of image scaling from patch-specific information, to clarify whether the performance gains in design 4 are due to shared prompting or the lack of image scaling?\n\nMathematical Proof for Row and Column Information Link: Could you include a more formal mathematical explanation or proof of how the information across rows and columns in the visual prompts is linked, ensuring that the inductive bias of neighboring pixels being more related than distant ones is preserved?\n\nTheoretical Guarantees on Convergence and Robustness: Can you offer theoretical insights or guarantees about the convergence, optimality, or robustness of the LOR-VP method, to complement the empirical results and ensure its reliability in diverse settings?\n\nClarifying the Contributions of Output Transformation vs. LOR-VP: Could you conduct additional ablation studies to more clearly separate the impact of the output transformation from the LOR-VP component, particularly to clarify why ViT models show a significant performance drop with ILM and FLM while Swin models do not?\n\nAnalysis of Sensitivity to Label Mapping in Different Architectures: Can you explore why ViT models seem more sensitive to label mapping methods compared to Swin models, and provide a deeper investigation into the factors causing this discrepancy?\n\nHandling Noisy or Adversarial Images: Can you include experiments testing LOR-VP's performance under adversarial attacks or in the presence of noise, to assess its robustness and ensure its reliability in more challenging or real-world scenarios?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Organic integration of LoRA and VP: The originality and novelty of this paper lie in its clever integration of Low-Rank Adaptation (LoRA) with Visual Prompting (VP), two previously established concepts, to create a highly efficient and effective approach for adapting pre-trained vision models. While LoRA has been used to reduce the complexity of model fine-tuning, and Visual Prompting focuses on task-specific adaptation through input modification, the paper's innovation is in combining these methods in a seamless way that enhances both parameter efficiency and model performance. By introducing low-rank matrix multiplications into the visual prompting process, LOR-VP allows shared and patch-specific information across the entire image, significantly outperforming existing methods in both speed and accuracy.\n\nExemplary clarity of reasoning:  In the preliminary study, the authors provide a well-structured and logical explanation for their design choices. They clearly demonstrate why both patch-specific and shared information in visual prompts are necessary by highlighting the limitations of existing methods that treat patches independently or focus only on peripheral areas. Additionally, their decision not to scale down the image emphasizes the importance of retaining maximum information for accurate model adaptation. This clear, step-by-step reasoning effectively justifies the development of LOR-VP, ensuring the method addresses these shortcomings while optimizing performance.", "weaknesses": "The preliminary study lacks rigor in controlling variables, as the impact of image scaling is not isolated from the role of patch-specific information. While design 4 (Patch-Same) outperforms others, the study does not definitively clarify whether its success is due to shared prompting across patches or the fact that the image is not scaled, leaving ambiguity about the true cause of the performance improvement. This undermines the ability to attribute the gains solely to patch sharing.\n\nIn the methodology section, the paper lacks formal mathematical proof detailing how the information across rows and columns in the visual prompts is linked, which leaves the assumptions of inductive bias vague. Additionally, while the low-rank matrix approach is intended to capture shared information, it does not explicitly guarantee that the natural relationship between neighboring pixels is preserved, and the exact nature of the associations formed between pixels remains unclear, weakening the justification for its effectiveness.\n\nThe method's performance relies heavily on empirical results without offering strong theoretical guarantees about convergence, optimality, or robustness in different settings, which could limit its broader adoption in critical applications.\n\nThe ablation studies comparing different output transformations lack clarity in distinguishing the contributions of the output transformation versus the LOR-VP component. Simply showing that LOR-VP outperforms other methods under the same output transformation doesn't clarify whether the performance gains are primarily due to the low-rank adaptation (LoRA) or the output transformation itself. Additionally, there is a noticeable performance drop for ViT models when using ILM and FLM, while Swin models do not exhibit this behavior. The authors fail to investigate or explain this discrepancy, leaving a gap in understanding why certain architectures are more sensitive to specific label mapping methods. A more detailed analysis of these interactions and a clearer separation of the contributions from each component are needed for a more rigorous assessment.\n\nThe paper lacks a thorough analysis of failure cases or edge scenarios where the LOR-VP method may struggle, such as on noisy or adversarial images. While the authors conduct robust experiments across multiple datasets and architectures, there is no exploration of how the method performs under conditions that deviate from the standard datasets, like adversarial attacks or high levels of image noise. For example, in Section 5.3, where robustness is discussed, the evaluation focuses on out-of-distribution generalization but does not account for adversarial robustness or resilience to noise, which are critical factors for real-world deployment. Without this analysis, it is unclear how reliable or stable LOR-VP would be in challenging environments, potentially limiting its practical use in more demanding applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)