Learning Visual Prompts for Guiding the Attention of Vision Transformers

## Abstract

Abstract Visual prompting infuses visual information into the input image to adapt models toward specific predictions and tasks. Recently, manually crafted markers such as red circles are shown to guide the model to attend to a target region on the image. However, these markers only work on models trained with data containing those markers. Moreover, finding these prompts requires guesswork or prior knowledge of the domain on which the model is trained. This work circumvents manual design constraints by proposing to learn the visual prompts for guiding the attention of vision transformers. The learned visual prompt, added to any input image would redirect the attention of the pre-trained vision transformer to its spatial location on the image. Specifically, the prompt is learned in a self-supervised manner without requiring annotations and without fine-tuning the vision transformer. Our experiments demonstrate the effectiveness of the proposed optimization-based visual prompting strategy across various pre-trained vision encoders.

## 1 Introduction

The attention mechanism [] in vision transformers leverages contextually relevant visual information from the entire input image to form embeddings for each token. This process dynamically adjusts the representation of each token, allowing the model to focus selectively on relevant areas, and integrates this context into the embeddings. Such context-aware embeddings lead to enhanced recognition of complex visual data []. In our study, we explore whether it is possible to direct the attention mechanism toward a specific input region by introducing and constructing a marker for pre-trained vision transformer models. We pursue this investigation to better *understand what kind of visual information directs the attention mechanism* of various pre-trained vision transformers and to *explore its potential for visual prompting*.

Visual prompting  [] refers to approaches that infuse visual information to the input images to adapt vision foundation models to new tasks.
Recent works introduce manual prompting techniques, such as placing colored circle or square markers[] on the image, or blurring regions surrounding the prompted location [],
and show they are effective in adapting CLIP vision encoder []
for fine-grained recognition tasks. These works hypothesize that these manual prompting techniques guide the attention of the model and therefore help the model with fine-grained recognition tasks.
However, using these manually engineering prompts rely on prior knowledge of biases (or emergent properties) formed in the model during training (It is reported that training data of CLIP contains these markers and blur effects [, ]).

In this study, we propose learning prompts (without fine-tuning the model) to guide the attention of various large models rather than manually designing them. We explore how we can find visual prompts that attract the attention of a pre-trained (and frozen) vision transformer (ViT []) through a self-supervised approach. *Thus, we would not rely on prior knowledge regarding dataset biases, and we can find prompts for any vision transformer trained on any dataset*. The optimization procedure alleviates the need for manual engineering or intuition to find the prompt.
Drawing inspiration from universal adversarial patches [],
and employing techniques to improve transferability such as utilizing a network prior  []
to generate adversarial perturbations [], we propose a self-supervised prompt optimization pipeline.
In addition to CLIP, we explore other ViT networks, such as DeiT [] and DINOv2 [, ],
each of which follows different training principles than CLIP (e.g. CLIP uses language supervision, while DINOv2 uses self-supervision).
Identifying visual prompts for DINOv2 is particularly important because it extracts significantly richer visual features from images and is recently emerging in the new generation of vision-language models [, ] due to CLIP’s visual limitations  [].
We observe that manual markers, like a red circle, do not effectively guide DINOv2’s attention, highlighting the need for an optimization-based approach for finding visual prompts.

Figure: Figure 1: Learned Prompt for CLIPs [], SigLIP [], DeiT [], DINOv2 []. In our framework, we learn a prompt to draw the attention of the model to a specific point where the prompt is applied. The prompt is optimized for each vision encoder model specifically to generalize across different images. The depicted image is taken from COCO [].
Refer to caption: extracted/5645791/Figs/demo.png

## 2 Related Work

Visual Prompting
Prompting has been extensively studied in the NLP community []. Recently, researchers have begun exploring the benefits of prompting in image recognition as well. This involves adding a learnable modification to images to guide models towards specific predictions [, , , , ].
These prompt-tuning methods  [, , , ] optimize a visual prompt that is appended and *fixed* to an input image.
For instance, this could involve appending optimizable pixel regions around the image.
By doing so, these methods enable the pre-trained model to adapt to new tasks without the need for retraining.
This modification is typically applied universally across different images.
Follow up approaches have been proposed to improve these visual prompts [, , ].
More recently, researchers have shown that manually crafted prompts, like a red circle, can effectively guide the attention of models like CLIP when trained on datasets containing similar markers
[].
Moreover, other biases, such as blurring, can be used for prompting  []. However, such attention-guiding prompts only work effectively on certain models. In this study, we propose learning prompts (without fine-tuning the model) to guide the attention of various models rather than manually designing them.

Adversarial Patch
The seminal work [] demonstrates that adversarial examples can be generated by altering just a few pixels in the input. This white-box attack [, , , ] method can deceive models by targeting a very small area.
Additionally, certain studies [, ] have successfully developed universal, transferable, and targeted adversarial patches.
These patches are typically placed on the primary object within images.
A related study [] found that model attention can be drawn towards adversarial patches designed to mislead classification results.
Our visual prompt can similarly be viewed as an adversarial patch, aiming to divert model attention away from its original focus, but for adapting model behavior towards useful tasks.

Transferability and Universality of Adversarial Perturbation The transferability of adversarial perturbation describes the phenomenon that perturbations crafted for one model can deceive another, even with a different architecture [].
Different from the transferability, the universality of adversarial perturbation is a property that the perturbation is still effective when it is added to different input instances [].
To improve transferability, we use generative adversarial perturbations  [].
A simple way to increase the universality is to include various input images when optimizing an adversarial perturbation [], which we also leverage in this work.
The transferability and universality of adversarial perturbation have also been studied [, , ].
These intriguing properties increase the threats in real-world applications.
However, how to leverage the two properties for good has not been explored in the community.

Figure: Figure 2: Overview of Self-supervised Prompt Optimization Framework: For a given image and a random position for the patch, the patch prior (a random noise) is first passed through a neural prior (an auto-encoder neural network). A desired mask is then applied to the patch to only partially cover the image and avoid losing data. The prompt is positioned on the target location and, after passing through the frozen Vision Encoder, the attention weights are extracted. The desired attention target values are calculated with a Gaussian distribution centered at the token corresponding to the target location. During training, the framework learns a prompt that minimizes the Kullback-Leibler (KL) divergence loss between the attention values of the CLS token and the target distribution.
Refer to caption: extracted/5645791/Figs/model_overview_2.png

## 3 Method

Our goal is to learn a patch such that if it is applied to any part of an input image, the corresponding tokens in the vision encoder would attract more attention, and therefore, the final representation of the image will be manipulated.
We propose a self-supervised method to learn such a patch, while the vision-encoder model is utilized in its frozen state and we only require a collection of images without the need for any labels.
The learning is about the patch’s RGB color space (three channels) while its shape is predefined, or alternatively, the shape mask can be learned as an additional fourth channel.
In the subsequent sections, we provide a detailed explanation of how this prompt is learned through the back-propagation of the self-supervised loss that accounts for the relation between the patch’s location and the attention it receives.

Input with Prompt.
We aim to discover a visual prompt (patch) that attracts the attention of a transformer-based vision encoder toward a specific location.
To achieve this, we work on the RGB image input denoted as $I\in\mathbb{R}^{n\times n\times 3}$ where $n$ is the number of pixels in one dimension of the image and $3$ indicates the three-dimensional color space.
The prompt, denoted as $\mathcal{P}$, is also in a similar RGB color space and of size $m$$\times$$m$ (the impact of prompt size on performance is investigated in section 4.1).
The designated coordinates $[i,j]$ within the image $0\leq i,j<n$ is then the exact pixel location where the visual prompt will be centrally inserted on.
By selecting coordinates only with criteria of $0<i\pm\frac{m}{2},j\pm\frac{m}{2}<n$, we ensure that the patch falls within the valid range, meaning that no part of the patch extends beyond the boundaries of the image after it is inserted.

The key point here is that by inserting we do not simply mean adding the values of the prompt pixel with those of the image pixel.
Rather, we aim to identify a general, universal patch that remains effective when applied to different images and locations, regardless of the underlying pixel values.
To achieve this, we replace the values of the corresponding image pixels with those of the patch pixels
$I[i-\frac{m}{2}:i+\frac{m}{2},j-\frac{m}{2}:j+\frac{m}{2}]=\mathcal{P}[:,:]$,
a process we refer to as insertion of patch into the image.
To ensure the transferability of the visual prompt, the placement of the patch is randomized across various locations within the image.
During the training phase, for an input image, we select $k$ random locations denoted by coordinates $[i,j]$ within the validity range criteria mentioned.
The patch is then inserted on each coordinate one at a time.
This process is repeated for all images $\mathcal{I}$ in the dataset, resulting in a set of modified input images $I_{\mathcal{P}}$ for all samples.
These modified images, with patches inserted at random locations, are then fed into the vision encoder.

Transformer-based Vision Encoder.
Within a transformer block with $L$ layers and $H$ attention heads, we denote the attention values as $A^{i}_{l}\in\mathbb{R}^{t\times t}$, where $A$ is the attention values of the $i^{th}$ head of the layer $l$ for an input image with $t$ tokens.
The output of the transformer encoder is typically a contextualized representation of the image tokens, where each token has been influenced by its surrounding tokens through the attention mechanism.
This representation, often obtained from a special token (e.g., the CLS token), can then be utilized for downstream vision tasks such as object recognition or image classification.
Our method leverages the attention values of the CLS token in the last layer, averaged over the heads of that layer: $\bar{A}_{L}[CLS,*]=\sum_{i=1}^{H}A^{i}_{L}[CLS,*]$.

Target Gaussian Map.
When the prompt is applied to the input image on the pixel position $[i,j]$, it overlays on one or more tiles with the center $[x,y]$, refer to
Fig. 2.
Assuming that the model divides an $n\times n$ pixel image to $t\times t$ image tokens with each tile having the pixel size $n_{t}$ ($n/t=n_{t}$),
the corresponding patch center $[x,y]$ can be derived from its pixel position on the image $[i,j]$ as: $x,y=(i/n_{t},j/n_{t})$.

To construct the target Gaussian map for the patch over the $t\times t$ token space, a 2D Gaussian distribution is employed.
The Gaussian map $\mathcal{G}(x,y)$ at location $[x,y]$ is represented by the probability density function of a Gaussian distribution $N(\mu,\sigma^{2})$, where $\mu=(x,y)$ denotes the mean (center) of the distribution,
and $\sigma$ is calculated from the Full Width at Half Maximum (FWHM), which in our experiments is set to the patch size in token space $m/n_{t}$: $\sigma=(m/n_{t})/(2\sqrt{2\ln(2)})$.

Neural Prior.
To learn the optimized prompt, we aim to avoid the expensive need to fine-tune the large-scale vision encoder by employing it solely in a frozen state.
In this way, the only parameters being updated through the training process are the pixel values of the prompt. However, a potential barrier we may encounter in efficiently achieving the optimal patch is the limited learnable variables within the pixel space [, ].
Thus we parameterize the patch input space with a neural network as in []. Such parameterization is also shown to improve the transferability of optimized perturbations [].
we use a neural prior $f$ starting from randomly initialized weights, that receives a random prior input $\eta\in m\times m$ , and outputs an initial prompt $\mathcal{P}_{prior}=f(\eta)$ which is then masked by a predefined shape mask
$\mathcal{P}=\mathcal{P}_{prior}*\mathcal{P}_{mask}$
and finally inserted centrally on the $[i,j]$ pixel of the input image $I$ to form the input to the vision encoder: $ViT(I_{\mathcal{P}})$.
Our neural prior $f$ employs a CNN-based architecture, in particular a U-Net with three layers of downsampling and three upsampling with
Sigmoid activation to ensure the values stay between $[0,1]$.
Leveraging shared spatial patterns among pixels in the patch enables effective optimization during training which facilitates efficient visual prompt learning [].

Figure: Algorithm 1 Learning Prompt with Predefined Shape

Objective Function.
Assuming that an image with a patch applied on it $I_{\mathcal{P}}=Insert(I,\mathcal{P},[i,j])$ is the input to the $ViT$, we define mean of last-layer attention values over attention heads of the vision encoder model as $\bar{A}_{l}=Attention(ViT(I_{\mathcal{P}}))$.
Our objective is to train the deep neural prior to output a prompt $\mathcal{P}$ such that it enhances the attention $\bar{A}_{l}$ at corresponding token at $x,y=(i/n_{t},j/n_{t})$ with $n_{t}\times n_{t}$ indicating the number of pixels in a token.
Having this objective in mind, we calculate the final self-supervised loss as follows:

$$ $\mathcal{L}(I,[i,j])=\mathcal{L}_{\text{KL}}(\bar{A}_{L}[CLS,*],\mathcal{G}(x, y))$ (1) $$

where $\mathcal{L}_{\text{KL}}$ is the KL-Divergence loss, Throughout the training process, the encoder remains frozen, and only the weights of the neural prior undergo updates as a result of the back-propagating of the defined self-supervised loss.

## 4 Experiments

In the following, we first present the learned prompts for CLIP-B/32 and CLIP-L/14, discussing several design considerations and their impacts.
We then evaluate the effectiveness of the prompts in the context of Naming Keypoints task. We take it a step further by learning prompts for SigLIP [], DeiT [], and DINOv2 [],
each of which either features a slightly different encoder architecture, or has a different training objective function, or varies in pretraining datasets.
Finally, we compare how the prompts impact their attention through layers.

Implementation Details and Settings.
All the vision encoders used are pretrained models available from publicly accessible libraries such as Hugging Face Transformers [] and PyTorch Hub [].
Unless specified, we use a learning rate of 1e-3 and batch size of 32 and train our framework for 10 epochs.
All experiments were conducted on a machine equipped with a single NVIDIA A100 GPU with 80GB of memory.

Datasets and Evaluation Metric.
Training: To train our neural prior, we use images from ImageNet [] which consists of millions of labeled images spanning one thousand classes.
However, as working with the full ImageNet is resource-demanding, we exploit a subset of it comprising of 10
categories according to [].
The limited coverage of label classes is not an issue for our work, as our method is not supervised and thus does not rely on specific label types.
Furthermore, the over 13k samples in [] offer a rich and diverse set of images, enhancing the generalizability of our self-supervised training.
Evaluation: For naming a keypoint task, we utilize CUB-200-2011 test set [] to evaluate our learned prompt.
This dataset contains 5794 bird images of different types, annotated with 15 body-part names and corresponding pixel coordinates as keypoints on the bird image.
We evaluate our method by positioning our learned prompt on these keypoints and then calculating the performance based on the percentage of correctly identified body parts.
To further evaluate our learned prompt, we test its performance also on the RefCOCO [], RefCOCO+[], and RefCOCOg[]
datasets which consist of images, annotated with bounding boxes around objects in it, each of which is paired with expressions.

Models and Baselines.
Specifically, CLIP-B/32 and CLIP-L/14 are employed for their robust zero-shot learning capabilities, leveraging large-scale vision-language pretraining.
SigLIP integrates image and language understanding in a synergistic manner.
DeiT and DINOv2 are utilized for their state-of-the-art performance in vision transformer architectures, with DeiT focusing on efficient training and DINOv2 providing self-supervised learning benefits.
The baseline for comparison involves using the cropped region over the object bounding box in the RefCOCO dataset family.
This approach isolates the relevant object from the surrounding context, allowing for a focused evaluation of keypoint identification accuracy.
Additionally, random location baselines are employed to assess the robustness of our method against random visual prompts.

### 4.1 Guiding Attention to Name Keypoints

We initiate our experiments with a basic prompt configuration, termed a vanilla patch, which is a simple filled square.
This patch undergoes no shape mask filtering during training, and the learned prompt is optimized RGB values.
We train patches of varying sizes for the CLIP-B/32 and CLIP-L/14 models.
Our experiments utilize prompt sizes corresponding to the model token dimensions.
As CLIP models split input images into tokens of different pixel sizes (base model: $32\times 32$, large model: $14\times 14$), our learned prompt of one-token size inherently possesses varying pixel dimensions.

Table 1 presents the visualizations of the prompts learned for both models at different sizes.
We select sizes proportional to the tokens: 1, 2, 3, and 4 times larger for CLIP-L/14.
For CLIP-B/32, due to the substantial token pixel size, we opt for prompts that are 1, 1.5, 2, and 2.5 times larger to avoid excessive pixel coverage by the patch.
We report the accuracy of keypoint naming on the CUB dataset.
The results indicate that for CLIP-B/32, the prompt size equal to 1 token yields the best performance, whereas for CLIP-L/14, the 3-times larger prompt achieves the highest accuracy.
Looking into this deeper, this suggests that optimal accuracy is achieved when the prompt covers approximately the same area size on the image ($32\times 32$ and $42\times 42$) for both models.
This may be because a larger patch, while better at manipulating the image, also covers more of it, leading to information loss and lower performance.

**Table 1: Vanilla Patches and Scale Constraints. The tile sizes of the CLIP-B/32 and CLIP-L/14 models are $32\times 32$ and $14\times 14$, respectively. We defined the dimensions for the vanilla patch (a simple filled square prompt) accordingly. For CLIP-B/32, the patch sizes are scaled proportionally to the model’s input token at $[1,1.5,2,2.5]$ times larger. For CLIP-L/14, the patch sizes are $[1,2,3,4]$ times larger than the token. These scales are chosen to ensure the patch is sufficiently large for effective learning without excessively covering image information. These prompts are evaluated on the keypoint naming task using the CUB dataset.**
|  | (a) CLIP-B/32 | (b) CLIP-L/14 |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Prompt Size | 32x32 | 48x48 | 64x64 | 80x80 | 14x14 | 28x28 | 42x42 | 56x56 |
| Visualization |  |  |  |  |  |  |  |  |
| CUB accuracy | 10.97 | 9.85 | 9.71 | 7.35 | 12.86 | 16.04 | 17.41 | 13.68 |

Beyond vanilla patch towards using shape priors: Initially, we employed a filled square, but previous studies imply that the geometric shape of the patch influences its effectiveness
[, ].
To investigate the influence of prompt shape on performance in naming keypoints task for CLIP models, we employ hollow square and circle as predefined shape masks.
We define a thickness factor as $\lambda=$ $\frac{inner~{}Diameter}{outer~{}Diameter}$ as shown in Figure 3 and investigate the performance for different thicknesses together with different patch sizes, again proportional to the model’s token size.

Figure: (a) CLIP-B/32. Square
Refer to caption: extracted/5645791/Figs/ratio_thickness/base_square_.png

What we observe from the performances in Figure 3 is that the circle-shaped prompt yields higher accuracy compared to the square-shaped prompt. For CLIP-B/32, the circle prompt achieves the highest accuracy of 11.51, and the best square prompt yields slightly lower accuracy. For CLIP-L/14, the difference is significant: the circle prompt reaches an accuracy of approximately 30.5, whereas the square prompt only achieves about 20.5. The comparison of patch sizes reveals findings consistent with the vanilla patch results. For CLIP-B/32, the optimal patch size remains the same as the default size for one token. In contrast, for CLIP-L/14, the highest accuracy is obtained with prompt three times larger than one token of the large clip model, the size covering a similar number of pixels. The visualizations of the learned prompts with the highest performance are depicted in Table 2. We observe that the red color for CLIP-L/14 and the pink color for CLIP-B/32 are predominantly present in the learned prompts. Interestingly, for CLIP-L/14, when the prompt shape is a circle, it learned a solid red color which confirms the findings in [].

**Table 2: Visualization of optimal prompt for various CLIP encoders. Here we visualize the best-performing square frame and circle-shape prompts from Figure 3. C stands for Circle and S for Square. Colors of red and pink are predominant for CLIP-B/32 and CLIP-L/14, respectively. It is noteworthy that the optimal prompt for CLIP-L/14 is a red circle. Even when we use a square shape prior, a red circle emerges inside the square. Previously, [] tried various markers using intuition and identified a red circle as an effective prompt. Interestingly, the marker seems to be the optimal marker for guiding the attention of CLIP-L/14. Though for CLIP-L/14 the red circle is not optimum. Thus it may be an emergent property arising from scale.**
| Encoder/Shape | CLIP-B/32, C | CLIP-B/32, S | CLIP-L/14, C | CLIP-L/14, S |
| --- | --- | --- | --- | --- |
| visualization |  |  |  |  |

Now that we have developed our prompt and explored the design configurations, we are interested in evaluating its performance compared to baseline methods on different datasets.
To this end, we train our method on the ImageNet dataset and test it on the CUB and RefCOCO datasets. These datasets are new to the prompt and have not been seen during the training phase.
Additionally, they offer the advantage of annotated image locations with names of specific body parts or objects for CUB and RefCOCO, respectively.

We define the baselines as Random: Randomly selected areas.
Crop: Removing the area around the bounding box for RefCOCO and defining a square similar to the prompt size for CUB, then cutting out that area.
Blur: Similar to cropping, but with the additional step of blurring the outer area using the averaging method with a kernel size of 5.

As indicated by the results presented in Table 3, the Crop method exhibits superior performance in RefCOCO tasks, but its effectiveness diminishes when applied to the CUB dataset.
In contrast, the Blur method shows notable improvements in RefCOCO tasks.
Our observations suggest that RefCOCO object descriptions often focus solely on the target object rather than relying heavily on scene context.
This tendency explains the success of both the Blur and Crop methods in this dataset.

However, in the CUB dataset, accurately recognizing animals can be challenging for the vision encoders, making it difficult to detect specific body parts.
Despite this challenge, our learned prompts generally outperform other baselines in the CUB dataset.
In RefCOCO tasks, our method consistently achieves second place after the effective Crop method.
These findings underscore the effectiveness of our approach, especially when the context of the image is important.

**Table 3: Performance comparison of different methods across various datasets. Bold numbers indicate the best performance, while underlined numbers denote the second-best performance. The Crop method demonstrates superior performance in RefCOCO tasks but exhibits reduced effectiveness in the CUB dataset. Conversely, the Blur method performs significantly better in RefCOCO tasks. Our observations reveal that object descriptions in RefCOCO do not always rely heavily on scene context, explaining the success of both Blur and Crop methods. However, in the CUB dataset, where context is crucial for proper animal detection, losing context leads to decreased model performance.**
| Encoder | Prompt | CUB | RefCOCO | RefCOCO+ | RefCOCOg |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| K2N | TestA | TestB | Val | TestA | TestB | Val | Test | val |  |  |
|  | Random | 8.2 | 36.9 | 39.5 | 37.8 | 36.9 | 39.7 | 37.9 | 39.5 | 39.1 |
| CLIP-B/32 | Crop | 15.6 | 56.6 | 47.7 | 52.6 | 60.4 | 51.7 | 55.9 | 60.3 | 59.5 |
| Blur | 7.4 | 39.6 | 41.1 | 40.0 | 39.5 | 41.2 | 40.0 | 41.3 | 41.1 |  |
| $32\times 32$ | 11.39 | 37.7 | 40.9 | 39.3 | 38.5 | 40.2 | 38.5 | 39.1 | 39.0 |  |
| CLIP-L/14 | Crop | 18.5 | 58.8 | 47.8 | 53.5 | 63.5 | 51.4 | 57.5 | 61.2 | 60.6 |
| Blur | 11.0 | 42.5 | 41.6 | 41.2 | 43.3 | 41.2 | 41.9 | 42.9 | 43.2 |  |
| $42\times 42$ | 29.7 | 49.5 | 43.9 | 46.7 | 51.3 | 46.2 | 48.1 | 50.4 | 49.0 |  |
| SigLIP | Crop | 19.6 | 62.6 | 51.3 | 57.9 | 66.8 | 56.7 | 62.3 | 69.2 | 68.5 |
| Blur | 13.1 | 42.9 | 41.3 | 41.8 | 43.6 | 43.0 | 42.8 | 44.2 | 44.5 |  |
| $48\times 48$ | 30.8 | 43.7 | 43.5 | 42.9 | 45.1 | 43.8 | 44.0 | 45.1 | 45.0 |  |

### 4.2 Investigating Attention Dynamics in Various Vision Encoders

Various vision encoder models are trained on diverse datasets, leading to behaviors that may differ from CLIP models.
More importantly, these vision foundations use different training paradigms.
CLIP vision encoder is trained with language supervision, while DINOv2 is trained by self-supervision using visual information only.
Therefore, during training is no supervision signal for the model to learn that colored markers (such as red circles) signify the importance of the region the marker is placed.
Additionally, their architectural details might vary slightly, e.g., SigLIP has no CLS token and instead averages over tokens by an attention pooling mechanism.
As a result, the optimal prompt to redirect their attention can also be different.
To investigate this, we use a similar self-supervised training approach to learn prompts of SigLIP, DeiT, and DINOv2 as well.

Figure 5 shows the visualization of the learned prompt for each vision encoder. The different appearance of the visual prompts confirms that there is no single prompt that is universally optimal for all encoders.
We apply the learned prompts on to three different locations on an image and compare the heatmap visualizations of the prompted image with the original, unprompted image.
For almost all encoders, the attention heatmap shifts towards the prompt location, demonstrating the effectiveness of our prompt in manipulating the attention of the corresponding visual encoder models.

Figure: (a) CLIP-B/32
Refer to caption: extracted/5645791/Figs/attGains/att_clipB.png

Figure: Figure 5: Learned Prompt for Different pretrained Vision Transformers. The optimal visual prompts are not the same and each model has its own unique pattern. The prompt is optimized to generalize across images. Comparing attention heatmaps of the original (on left) and prompted images (on right) reveals how effectively the prompt directs attention to specific locations. The prompt is optimized for each ViT (CLIP-B/32, CLIP-L/14, SigLIP, DeiT, and DINOv2) separately, and is optimized over 20k random samples from ImageNet []. The depicted image is taken from MS COCO [].
Refer to caption: extracted/5645791/Figs/att_heat_map.png

The effect of the optimized prompt on attention across layers:
We examine the attention averaged values of tokens in the area overlaid by the prompt before and after its application for over 1000 images from MS COCO while the prompt location is random. By analyzing attention values across layers, we can assess the prompt’s impact. Since attention values are relative, we define Attention Gain as the ratio of the difference of averaged attention values of tokens influenced by the prompt to their original attention values. Figure 4 illustrates the attention gains across different models. The blue line represents the gains for each model’s optimized prompt, while the orange line shows the gains for a simple red circle marker prompt.

Red circle does not guide attention of DINOv2 and DeiT:
From Figure 4 we can see that the unique prompts for DeiT and DINOv2 draw significantly larger attention to themselves compared to the red circle prompt, showing the power of our learning framework.
This could confirm the suggestion that the presence of red circles in the training data of CLIP models and SigLIP has made them particularly sensitive to this predefined feature [], which is not the case for other vision encoders such as DeiT and DINOv2.

The significance of optimizing prompts for self-supervised models such as DINOv2:
To further evaluate the effectiveness of our prompts, we use the MLLM (Multimodal Large Language Model) introduced in [], which uses an Interleaved Mixture-of-Features approach to spatially leverage interleaving CLIP and DINOv2 visual tokens after an adapter.
In Figure 6, we compare [] model’s performance with and without our unique CLIP-L/14 and DINOv2 prompts applied to the image input to see if it improves the question answering responses of their model on proposed MMVP (Multimodal Visual Pattern) Benchmark.
The examples demonstrate that our prompts enable the model to generate more accurate answers, which means they create improved embeddings.
This indicates that our attention-guiding prompts have promising applications in a variety of vision tasks that can be explored in future works.

Figure: Figure 6: Prompts for the new generation of Vision Language Models: New vision language models are moving beyond clip vision encoders towards using richer vision encoders such as DINOv2. Therefore it becomes progressively necessary to identify visual prompts for the new generation of models. This figure depicts an example to show the potential of using the optimized prompts in the new generation of vision-language models, in this case, LLaVA+MoF [] on examples from the MMVP dataset.
Refer to caption: extracted/5645791/Figs/eyes_.png

## 5 Conclusion

In this work, we proposed a self-supervised optimization-based visual prompting technique for guiding the attention of vision transformers, thereby avoiding the limitations of manually crafted prompts. Our method demonstrated the ability to guide the attention of various vision transformer models, such as the CLIP family, SigLIP, DeiT, and DINOv2, without requiring prior knowledge of dataset biases and without fine-tuning the models. The transferability of the prompt across different images was accomplished by leveraging a deep network prior. Our experiments confirmed that the learned prompts successfully redirect the attention of not only the CLIP family of models which are trained with language supervision, but also the purely self-supervised models such as DINOv2. As new vision language foundation models are leveraging the self-supervised vision encoders for their superior ability to extract visual features, the proposed optimization-based proves to be helpful for prompting upcoming models.

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 3
presentation: 2
questions: Please refer to the Weaknesses section for further details.
soundness: 1
strengths: - The paper is easy to follow and understand.
- This method can be applied to different pre-trained ViT, regardless of their pre-training supervision methods, and does not require annotation or fine-tuning of the ViT.
summary: This paper focuses solely on learning prompts through self-supervised learning, which can be added to any input image to redirect the attention of the pre-trained ViT to its location. It achieves this by minimizing the KL divergence between the attention of the CLS token and the target distribution.
weaknesses: - The paper lacks essential details and descriptions, which may lead readers unfamiliar with the preceding work to struggle with understanding what is being tested or implemented. For example, the methodology for testing on the CUB dataset is not clearly outlined, and the significance of "K2N" on line 478, which stands for keypoint to name, is not explained.
- Building on the concept that a "red circle" can direct CLIP's attention to a specific area, as demonstrated in prior work by Shtedritski et al. (2023), the paper's technical contribution is limited, and its effectiveness is also open to debate:
    1. It primarily applies the phenomenon of the "red circle" to other pre-trained Vision Transformers through learnable prompts. However, the experiments involving language-supervised methods lack baselines for the "red circle" and cropping techniques. Moreover, the performance comparison with the "red circle" from the referenced paper [https://arxiv.org/pdf/2304.06712] appears to show a decline rather than an improvement.
    2. In the DINOv2 experiments, the paper shows examples from MMVP for illustration but does not provide quantitative metrics.
    3. The paper combines the phenomenon of the "red circle" with visual prompt learning, which is incremental. Moreover, its effectiveness has not been well demonstrated.
- The application scenarios are limited: the phenomenon of the "red circle" is quite interesting and helps us better understand CLIP. This paper focuses on applying this phenomenon to more pre-trained Vision Transformers. However, the method of using circles necessitates manually annotating areas of interest on images, making it impractical for large datasets. Consequently, effectively applying this phenomenon to downstream tasks is challenging. If possible, it should be stated how the proposed method can be applied to downstream tasks.
- There is a lack of suitable baselines and quantitative comparisons. In the absence of an appropriate baseline, it is challenging to demonstrate that the proposed method improves upon previous approaches. Since the paper’s contribution is to extend this phenomenon to other ViTs, it is essential to conduct controlled ablation experiments on the MMVP-VLM dataset to show that the proposed method is more effective than the simple "red circle" approach.
- Minor: The presentation of the figures needs improvement; for instance, in Figure 7, the text for location 2 3 is obscured.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: 1. When applying multiple visual prompts simultaneously, do the attention maps exhibit similar patterns of change as observed with single prompts?  
2. How does the performance compare quantitatively to using a simple red circular frame as a visual prompt?  
3. Could you consider using attention gain as a training objective to enhance the model's ability to follow visual prompts with LoRA (a efficient training method)? This could provide an interesting baseline comparison.
soundness: 2
strengths: 1. The approach is model-agnostic and does not require fine-tuning, making it versatile across different models.  
2. The method demonstrates strong generalization capabilities across various models and datasets.  
3. The presentation is clear and easy to understand.
summary: This paper introduces a self-supervised learning approach for generating visual prompts that guide the attention mechanism of Vision Transformers (ViT). The method utilizes a neural network prior and optimizes the prompt pixels through backpropagation to enhance attention at corresponding positions in the visual encoder. The approach is evaluated on the CUB and RefCOCO datasets, showing improved performance compared to baseline methods.
weaknesses: 1. The paper lacks sufficient justification for the advantages of the approach compared to directly collecting and fine-tuning with annotated visual prompt data, particularly in terms of performance and efficiency benefits.  
2. There are some typos: some citation formats need correction; Table 2 shows inconsistent decimal places across the results; the ordering in Figures 4 and 5 appears to be incorrect.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: In light of these weaknesses, I would suggest the authors address the following: 1. How does the method perform on real-world datasets with complex and diverse scenes? The paper’s focus on curated datasets like CUB and RefCOCO raises questions about robustness on uncurated, real-world datasets with more complex scenes. Such environments may demand attention guidance that adapts beyond controlled settings, so further validation in these contexts would be beneficial. 2. Could the approach be generalized to non-transformer vision models? As the study is limited to transformer-based models like CLIP and DINOv2, testing with other architectures (e.g., CNNs) could clarify whether the learned visual prompts are uniquely suited to transformers or if they have broader applicability across different model types. 3. How can we ensure the accuracy of attention guidance? Since the model’s "attention" relies on pre-trained representations, this approach may lack robustness across different test set distributions (e.g., open-set scenarios). The work’s effectiveness hinges on accurate attention to guide model behavior, but if the attention is inconsistent, it could potentially mislead the model, impacting performance.
soundness: 2
strengths: The approach appears to be adaptable, allowing self-supervised learning of visual prompts for various vision encoders without needing labeled data. This flexibility is particularly valuable for adapting vision transformers across tasks and models without costly retraining or fine-tuning.
summary: This paper proposes a self-supervised optimization approach for designing visual prompts to guide the attention of vision transformers (ViTs) toward specific regions in an image. Instead of manually creating visual markers, the method involves learning visual prompts specific to each vision encoder, enhancing its focus without altering its architecture. Experiments on CLIP, DINOv2, and other ViTs show that these prompts effectively influence attention distribution, improving model performance on tasks like keypoint naming and object recognition.
weaknesses: There are a few concerns for the algorithmic design: 1. Although the method is intended as a self-supervised approach, its performance is primarily evaluated using annotated datasets like CUB (for keypoint detection) and RefCOCO (for object localization), where specific body parts or objects are pre-labeled. This reliance contrasts with the self-supervised nature of the learning process, potentially limiting practical utility. For instance, if access to labeled datasets is restricted or unavailable in a particular domain, the evaluation and validation of the learned prompts could become challenging, affecting the applicability of the approach in real-world, unlabeled environments. 2. The evaluation is performed on datasets with relatively controlled settings (e.g., images with birds or single annotated objects). In more complex, natural scenes where multiple objects and overlapping contexts are present, the effectiveness of learned prompts could degrade. Without sufficient evaluation on diverse, unlabeled datasets, there’s a risk that the method may not generalize effectively to scenarios with complex visual contexts. This gap highlights a limitation in validating the approach’s robustness and universality outside curated datasets. 3. The learned prompts are customized for individual transformer-based architectures (e.g., CLIP, DINOv2), which may not translate effectively to other model types or even to transformers with slight architectural variations. This specificity can limit cross-model generalization, requiring separate training of prompts for each new architecture. For broader applicability, it would be beneficial to explore if and how prompts could be designed with more universal features or optimized independently of model architecture.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: See the comments below.
soundness: 2
strengths: 1. The structure of this paper is complete.
2. The content of this paper is easy to follow.
summary: This paper explored the possibility of guiding the attention mechanism to focus more on a region of interest by introducing visual cues to the input while avoiding intervention on the internals of the ViT. However, I have some concerns as follows:
weaknesses: 1. The Introduction is confusing. There is no logical relationship between the three paragraphs of the introduction, and it is unclear what the author is trying to do within the three paragraphs. How do existing methods do it? What problems do they have? What is the difference between the proposed method and the above methods? What is different and innovative about the proposed method?
2. In related work, what is the relationship between the three parts of the introduction? What problems do existing methods have that should be further elaborated?
3. The proposed method is too simple. It feels like a patchwork of existing methods. Compared with existing methods, I don’t know where the core innovation of the proposed method is.
4. The experimental results are unreliable. First, there is a lack of comparison with the most advanced methods. Second, the analysis is insufficient for ablation studies. It is unclear why each of the proposed components can bring performance improvements.

Meta-review: The paper received four expert reviews with 3x reject and 1x borderline reject ratings. Overall, the reviewers are concerned about limited technical novelty, limited effectiveness, limited application scenarios of the method, and unjustified results. The authors are suggested to rethink the formulation of the problem domain and improve the manuscript based on the reviewers' comments.

Decision: Reject