Fine-Grained Visual Prompting
Authors: Lingfeng Yang
Yueze Wang
Xiang Li
Xinlong Wang
Jian Yang
Venue: NeurIPS 2023 poster (2023)
Keywords: visual prompting
zero-shot
visual language model
referring expression comprehension
TL;DR: A zero-shot architecture using fine-grained visual prompting for referring expression comprehension and object grounding.

Abstract: Vision-Language Models (VLMs), such as CLIP, have demonstrated impressive zero-shot transfer capabilities in image-level visual perception. However, these models have shown limited performance in instance-level tasks that demand precise localization and recognition. Previous works have suggested that incorporating visual prompts, such as colorful boxes or circles, can improve the ability of models to recognize objects of interest. Nonetheless, compared to language prompting, visual prompting designs are rarely explored. Existing approaches, which employ coarse visual cues such as colorful boxes or circles, often result in sub-optimal performance due to the inclusion of irrelevant and noisy pixels. In this paper, we carefully study the visual prompting designs by exploring more fine-grained markings, such as segmentation masks and their variations. In addition, we introduce a new zero-shot framework that leverages pixel-level annotations acquired from a generalist segmentation model for fine-grained visual prompting. Consequently, our investigation reveals that a straightforward application of blur outside the target mask, referred to as the Blur Reverse Mask, exhibits exceptional effectiveness. This proposed prompting strategy leverages the precise mask annotations to reduce focus on weakly related regions while retaining spatial coherence between the target and the surrounding background. Our **F**ine-**G**rained **V**isual **P**rompting (**FGVP**) demonstrates superior performance in zero-shot comprehension of referring expressions on the RefCOCO, RefCOCO+, and RefCOCOg benchmarks. It outperforms prior methods by an average margin of 3.0\% to 4.6\%, with a maximum improvement of 12.5\% on the RefCOCO+ testA subset. The part detection experiments conducted on the PACO dataset further validate the preponderance of FGVP over existing visual prompting techniques. Code is available at https://github.com/ylingfeng/FGVP.

Fine-Grained Visual Prompting
Lingfeng Yang', Yueze Wang", Xiang Li Xinlong Wang", Jian Yang
'Nanjing University of Science and Technology BBeijing Academy of Artificial Intelligence, "Nankai University
fyanglfnjust, cejyang)onjustt edu.. en, (yzwang, wangcinloonghlaasi.ee xiang. li. implus@nankai. cn
Abstract
Dision-Langura Models (VLMs), such as CLIP, have demonstrated impressive zero-sho shot transfer capabilities in image-level visual perception. However, these models have shown limited performance in instance-level tasks that demand precise localization and recognition. Previous works have suggested that incorporating visual prompts, such as colorful boxes circles, can improve the ability of models to recognize objects of interest. Nonetheless, compared to language prompting, visual prompting designs are rarely explored. Existing approaches, which employ coarse visual cues such as colorful boxes or circles, often result in sub-optimal performance due to the inclusion of irrelevant and noisy pixels. In this paper, we carefully study the visual prompting designs by exploring more fine- tine-grained mark- ings, such as segmentation masks and their variations. In addition, introduce new zero-shot framework that leverages pixel-level annotations acquired from a generalist segmentation model for fine-grained visual prompting. Consequently, our investigation reveals that straightforward application of blur outside the target mask, referred
the Blur Reverse Mask, exhibits exceptional effectiveness.
This proposed prompting strategy leverages the precise mask annotations to re- duce focus weakly related regions while retaining spatial coherence between the target and the surrounding background. Our Fine-Grained Visual Prompting (FGVP) demonstrates superior performance in zero-shot comprehension of ferring expressions on the RefCOCO, RefCOCO+, and RefCOCOg benchmarks. It outperforms prior methods by an average margin of 3.0% 4.6%, with a maximum improvement of 12.5% on the RefCOCO+ testA subset. The part detection experiments conducted on the PACO dataset further validate the prepon- derance of FGVP over existing visual prompting techniques. Code is available at tttps://gtt thub. com/ hylllygggnnnngnnne
Introduction
The usage of Vision-Language models (VLMs) [42, 1, 300 has become increasingly prominent in various vision-related tasks, largely due to their notable impact. These models benefit from vast amount of training data and parameters, demonstrating powerful performance in the training of fundamental visual model backbones [34, 15, 15, Moreover, the transfer learning potential of VLMs has been employed in tasks such open vocabulary detection [20, 64], visual grounding [32, 37], and image editing [6. 5]. etc.
In the above-mentioned works, the VLMs generally require additional training, to adapt to each specific downstream task. However, the utilization of off- thf-hhh- Vision-Language Models (VLMs)
"Corresponding authors.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Keypoint [AI]
Colorful Crop
[A2]
Crop [P]
Raw Image
Coarse Prompting
Box [BI]
Colorful Box [B2]
Grayscale Reverse [B3)
Blur Reverse [B4]
Circle [C1]
Colorful Circle [C2]
Grayscale Reverse Circle [C3]
Blur Reverse Circle [C4]
Contour [DI]
Colorfill Mask [D2]
Grayscale Reverse
Blur Reverse Mask [D4]
[D3]
Line based [+
Color-based [+2]
Reversc-based [+3][-4]
Figure I: A Summary of visual prompts with the caption "elephant on the left".
for leveraging their inherent image-text understanding, without additional tuming, remains largely unexplored. A notable challenge lies in the low sensitivity of VLMs to perform object spatial localization due to the presence of large amount of weakly related background noise in the images [68, 35]. The common practice involves cropping the region of interest [10, 62] to obtain zoomed-in visual representation of single object, at the expense of discarding valuable global information. As a result, existing training-frit free approaches for classification lack a comprehensive understanding of contextual information and spatial awareness across different objects.
In recent studies, CPT [61] and ReCLIP [50] have leveraged visual prompting techniques to establish relations between partial instances. Visual prompting is technique used in image- langgala tasks, where markers such colorful boxes or circles are added directly onto an image highlight specific targets. By utilizing appropriate visual prompts, the attention of VLMs can be effectively directed toward the desired targets while preserving the global context. Moreover, a study by RedCircle [49] demonstrated that drawing red circles enclosing the object on the entire image can effectively distinguish instances, where the circles correspond to inscribed ellipses derived from proposal boxes. This discovery suggests that Vision-Languaal understand the local object within an overall image. Therefore, a specifically designed visual prompt has the potential to explicitly invoke this capability of VLMs, thereby benefiting various tasks.
Models (VLMs) may possess the inherent ability to
Despite the interest in visual prompting techniques, their unique designs have yet be fully explored. The current approaches only rely on coarse markers like colorful boxes or circles, which can introduce ambiguity and fail highlight the intended instance accurately. this paper, we address this issue by systematically organizing and investigating various forms of visual prompting. Further, propose refined technique called mark each target precisely, thereby enhancing zero-shot performance.
Visual Prompting (FGVP), which utilizes semantic masks to

---

M
SAM
FGVP
VLM
2 glass bowl
Detector
8 raspberries
apple
apple the left
apple middle
Figure 2: Structure of fine-grained visual prompting with box proposals from a detector.
In research, examine different visual prompting designs showcased in Fig. 1, including those utilized in previous studies such as CPT [61]. ReCLIP [50], and RedCircle [40]. To provide an overview of previous research, summarize their strategies in Table 1. It is important to note that some approaches achieve their best results through prompt ensembles or post processing techniques, which are discussed extensively in Sec. 3. Despite the progress made in visual prompting, none of the existing methods have fully explored the use of more precise semantic masks as fine-grained markers, rather than relying on coarse boxes circles. To verify the effectiveness, we begin by exploring the upper bound of these prompts. Our methodology assumes perfect accuracy of all annotations applied, ensuring that images are marked with ground-truth information, namely bounding boxes and masks. We confirm that single Blur Reverse Mask [D4], which involves blurring all content outside the instance mask, significantly outperforms other types of visual prompts on multiple datasets. This indicates the effectiveness of our fine fine-grained visual prompt design.
In practical scenarios where ground-truth annotations are unavailable, there are two primary approaches for obtaining fine- grained visual markings. The first ap- proach based on utilizing object pro- posals predicted by pretrained detec- tor, following similar paradigm em- ployed in MAttNet [62]. This approach commonly employed in zero-shot clas- sification tasks [61, 50, 491. To generate fine-grained masks, we employ Segment
Existing Method
Visual Prompt
Post Processing
MA#Net [62], UNITER [10]
CPT [61]
A2
ReCLIP [50] RedCircle [40]
P| 84
Relations [50]
C1
C3 C4
Subtraction [40]
Table 1: The type of Visual Prompt equipped in previous works related the referring expression comprehension task. denotes ensemble results of multiple prompts.
Anything Model (SAM) [26]. which takes the aforementioned bounding boxes (refer to Fig. 2) as input prompts. Additionally, we propose a second approach that eliminates the dependency on detector for prompt candidates. Our approach establishes viable zero-shot pipeline by solely lever- aging SAM, irrespective of specific tasks. Specifically, we prompt SAM with grid-wise keypoints and subsequently apply non-maximum suppression (NMS) operation to obtain proposals with semantic masks. These fine- -re-ggained masks can then be employed for zero-shot classification (see Fig. 3).
Our contributions can be summarized as follows:
We propose Fine-Grained Visual Prompting (FGVP), employing Blur Reverse Masks to enhance the semantic localization capability of Vision-Language models such as CLIP (Fig. 1). To the best of knowledge, highlighting their remarkable effectiveness zero-shot image-text comprehension.
the first explore the of Blur Reverse Masks visual prompting,
We are the first provide comprehensive exploration of visual prompt formats, including crop, box, circle, mask with different line markings, colored masks, grayscale, and Gaussian blurring. Additionally, thoroughly analyze and evaluate the impact of associated auxiliary attributes such as blur deviation, color, etc., ensuring a comprehensive understanding of their effects.
Our proposed Fine- Grained Visual Prompting (FGVP) achieves state-of-the-ee (SOTA) zero-shot results on the referring expression comprehension task, surpassing ReCLIP and RedCircle by an average of 4.6% and 3.0% accuracy, respectively, across RefCOCO, RefCOCO+, and RefCOCOg benchmarks. Notably, our zero-shot pipeline demonstrates superior part detection accuracy on PACO dataset compared to previous visual prompting methods.

---

2 Related Work
Vision-Language Models Large Language Models (LLMs), such as GPT-3 [7], GPT-4 [41], LLaMA [53], and PaLM [12], have demonstrated strong zero-shot transfer abilities in natural language processing. In recent years, vision-language Models (VLMs) that leverage image-text data pairs from the web have gained prominence in computer vision (CV) tasks. CLIP [42] and ALIGN [31] learn image-text alignment through contrastive learning. Furthermore, models like Flamingo [1] have shown impressive few-shot learning capabilities. BLIP-2 [30, 29] proposes joint multimodal task handling through transfer learning. Notably, VLMs have excelled in image-text generation demonstrated by DALL-E [46, 45]. However, instance-level tasks such referring expression comprehension and part detection typically require tuning of vision and text encoders in existing open vocabulary methods [20, 64] and image grounding approaches [32, 37). In contrast, this paper proposes a zero-shot architecture for instance-level tasks using off- off-the-shelf VLMs.
Prompt Engineering Prompt engineering is well-developed practice in NLP [43, 7]. Auto- Prompt [48] and CoOp [69] automate the generation of prompt templates, leveraging gradient-based methods instead of manual crafting. Language prompting is then extended to open vocabulary detection [14, 16] and segmentation [59]. While language prompting has been extensively explored, visual prompting has received less attention. Previous works [35, 22, 2, 4] use visual prompt tuning to adapt to VLMs. In terms of zero-shot solutions, CPT [61] introduces colorful boxes as markers on images, and RedCircle [49] demonstrates the effectiveness of a circle mark for visual attention during zero-shot classification. However, existing visual prompting methods lack fine- fine-graine grained precision. In contrast, we propose leveraging semantic masks derived from segment models like Segment Any- thing [26] for more precise visual marking. It is worth noting that CPT [61] also employs semantic masks as colored prompts, but only after cropping the region using bounding boxes. In our approach, we directly mark the fine-grained mask on the entire image to preserve the global visual content.
Image Segmentation Image segmentation is common task in computer vision [52], involving predicting region segment masks by labeling pixels. This task encompasses various sub-tasks such as instance segmentation [21, 13, 55], semantic segmentation [38, 67, 11], and panoptic segmen- tation [25, 25]. Recent advancements include SegGPT [56]. which facilitates performing diverse segmentation tasks through in-context visual learning. Another notable approach, Segment Anything Model (SAM) [26], introduces spatial prompts for segmenting arbitrary objects. SAM is trained using an extensive dataset of over billion segmentation masks. In our study, we employ the off-the-shelf SAM framework to generate precise semantic masks for fine-grained visual prompting.
Referring Expression Comprehension This task involves providing captions for specific objects images and localizing them with bounding boxes. Proposal-based methods use object detectors like Mask R-CNN [21] to detect instance regions. Each proposal is then cropped and used for subsequent categorization. Examples include MAtNNet [62], UNITER [10], OSCAR [33], and Vin [66]. Proposal-fre -fres methods, such as MDETR [23] and ViLT [24], train Vision Vision-Language Models end-to- end. Recent zero-shot methods [61, 50, 40] combine proposal boxes from MAttNet [62] with Vision- Language Models for image-caption matching. Our work follows this setup for fair comparisons.
Part Detection Part Detection is sub-field of object detection [21, 47, 17, 8], primarily focused on fine-grained classification [65, 54] and human parsing [58, 18] that require precise region localization. Notable datasets include CUB-200 [54] for annotated bird parts, LIP [19] for semantic human part labels, PASCAL-Part [9] for common objects, and PACO [44], which introduces a comprehensive benchmark with jointly annotated part masks. Previous zero-shot works typically detect objects and parts using pre-annotated visual attributes to transfer the knowledge [28]. In contrast, we leverage the
transferability of Vision- Dision-Languuge Models to perform part detection in this study.
3 Method
In this section, we begin by presenting a comprehensive overview of the visual prompting pipeline for zero-shot object recognition and part detection tasks. Subsequently, we delve into the details of our Fine-Grained Visual Prompting (FGVP) architecture. Finally, we examine the effectiveness and rationale behind our design.

---

3.1 Framework
The zero-shot framework takes as input an image E 3 .N box proposals E N and M caption texts T E S2Mr {t' t t..,,t, J, where Ω denotes the set of textual sentences. The goal is to find the best matching image-text pairs. A common practice cropping or RolAlign [21] according to $. However, with the introduction of visual prompting, one can mark the regions on the whole image, which highlights each target instance while keeping the background knowledge. For simplicity, we regard cropping as an ordinary type of visual prompting. for VLMs can be generated as:
get each image imput by
Then the image input I E R x3N (1/ R2p
(1) where VP concludes visual prompting, such as cropping, drawing colorful boxes or circles, or employing fine-grained masks, etc. Then the cosine similarity S = {s( (T)) N between each image proposal I and text 7 can be derived by the VLM, e.g., CLIP [42]. For the referring expression comprehension task, T is gathering of short sentences that describe the related instances in the image, and one needs to predict their referring location. Note that there exist some post processing techniques for this task, namely "Relations" [50] and "SSutraction" [49], respectively. The "Relations" considers spatial relations R = (r(()) ,2%) ]NNN between every two proposals, w.r.t left, right, above, below, bigger, smaller, and inside. Then the parsed final scores are updated as:
VP $).
(2) Besides, the "Subtraction"" deals with positive and negative text set to weigh down the score. By randomly sampling Q negative captions T that related to instances on the image, all negative scores are calculated as S _R Nx B6 t9). And then use it to penalize S:
S = ( eIa. (s(()) ( )),
)), E S,r E R).
S = { 1T))
E S 19), E S 5).
(3)
Q
Once the final scores are obtained, the referring region T can be given by:
( و ARGMAX,eEn st )""), E S}.
For the part detection task, the text is the object or part label names. Considering the general case, each part only appears in one region (all eyes of a bird are covered by a single bux). Thus we indicate post processing strategy for this task with the Hungarian algorithm [27], to find a bipartite matching between image proposals and part labels. Otherwise, the label predicted corresponding to each proposal is derived as:
-= ARGMAXX TM Ee t"),,
(5)
3.2 Fine-Grained Visual Prompting
Existing visual prompts primarily use proposal boxes from the detector. The Crop [P] and Colorful
Crop [A2] prompts involve extracting cutouts of the original image based on the location of the box. Prompting with Keypoint [A1] entails placing a small circle around the box center. The box-based [B*] prompts naturally use the box as the marking boundary. while the circle-based [C*] prompts are essentially similar [B,], except for replacing the box with its inscribed ellipse.
However, these prompts are too coarse to emphasize key targets. Marking inaccuracies introduce irrelevant information and lead to sub-optimal performance based on empirical evidence. We illustrate previous visual prompting approaches as coarse prompting on top of Fig. 1. In contrast, we investigate fine-grat -reiiea visual prompting using semantic masks, which can accurately highlight the target instance, reduce background interference. and retain global knowledge. Nonetheless, obtaining semantic masks may pose challenge when only proposal boxes are provided. To tackle this issue, we propose the use of robust image segmentation model called Segment Anything Model (SAM) [26] to perform class- agnostic segmentation based on the given boxes (Fig. 2):
M- = SAM(T, 4),
(6)
1 = MG)
(7)
where SAM denotes Segment Anything Model [26], and M E N denotes the semantic masks. We updated Eq. (1) with Eq. (6), (7) to produce fine-grained visual prompting.
In addition, We innovatively proposed a zero-shot classification pipeline, which does not require pre- pro-ppocessed box proposals but directly generates fine- finengrained markers. The key idea lies in that
5

---

M
SAM
RGVP
VLM
glass bowl
2 apple the left
raspberries
apple in the middle
Figure 3: Structure of fine-grained visual prompting with no box proposal. Masks are directly derived via SAM prompted by grid-wise keypoints.
SAM can propose an extremely exhaustive prediction of almost any object of parts on the images taken a grid-wise set of keypoints G E RR9x x2 as inputs (Fig. 3), where is the point number along side of the image. Then the mask is generated through SAM prompted by the grid-wise keypoints: M NMS(SAM(1, G));
(8) where NMS denotes Non-Maximum Suppression to filter out duplicate proposals. Notably, the regression box & is obtained by calculating the smallest box containing the object mask.
Following the FGVP illustration, we provide an overview of all visual prompting variants. In addition to the categorization based on box, circle, and mask, they can also be sorted based on different markers, such as line [+1], color [+2], grayscale [+3], and blur [+4]. Specifically, line-based methods utilize closure lines to prompt the image, while color-based prompting involves drawing colorful masks on the target. These two types can be classified as positive marking, which aims to highlight the target area. Conversely, the reverse-based prompting serves as a form of negative marking by grayscaling or blurring the background area, thereby reducing the impact of weakly related information.
3.3 Discussion
In fact, both part detection and referring tasks require mutual un- derstanding between instances and backgrounds. In the referring expression comprehension task, captions used describe the interrelationships between multiple objects. Similarly. in part de- tection, local part be hard classify regardless of the object. Successful performance of these tasks thus necessitates a network's s ability handle global information and accurately comprehend the relationships between instances. For this reason, implementing more precise visual prompting is helpful, typically with the Blur Re- verse Mask Prompting [D4]. This is because the web-scale dataset on which VLM trained contains a large amount of photography. These images prefer to employ "Bokeh" to blur the background and highlight the subject (Fig. 4). As a result, VLMs may have prior knowledge of visual prompting with background blurring.
Figure Photography from Internet with "Bokeh".
4 Experiments
In this section, we first evaluate individual visual prompting performance. Then, we compare FGVP with previous zero-shot methods the referring expression comprehension and part detection tasks to show our effectiveness. For more experimental analysis refer to the Supplementary Materials.
4.1 Dataset
We conduct the experiments several visual datasets, i.e., RefCOCO [63], RefCOCO+ [63], RefCOCOg [39], COCO [36], and PACO [44]. The RefCOCO, RefCOCO+, and RefCOCOg datasets subsets selected from COCO, containing bounding boxes and masks corresponding to captioned

---

Ground Truth RefCOCO
Referring Expression Comprehension
Visual Prompt
COCO
PACO
RefCOCOg ReCCOCO+
RefCOCo
RefCOCO+
RefCOCOg
Crop
70.9
38.5
35.2
45.3
Keypoint
52.3
39.1
36.9
39.6
43.8
46.7
47.9
48.9
Colorful Crop
64.2
35.7
37.1
41.9
58.0
48.2
49.0
57.0
Box
48.5
42.7
34.7
39.5
44.6
45.5
46.4
47.0
Colorful Box
34.4
37.2
23.9
23.4
22.7
35.4
30.7
30.8
Grayscale Reverse Box
42.4
37.4
34.4
35.9
44.5
45,9
44.0
48.4
Blur Reverse Box
62.1
39.2
47.9
51.8
63.6
48.8
51.4
54.1
Circle
48.9
42.6
43.2
49.3
56.3
48.9
51.7
54.6
Colorful Circle
36.1
37.2
29.9
29.8
24.5
40.7
37.1
37.9
Grayscale Reverse Circle
42.9
36.6
36.9
38.2
47.8
46.2
50.3
Reverse Circle
58.1
49.2
49.3
Contour
47.3
41.0
38.7
46.1
45.0
46.3
Mask
41.1
43.7
29.9
29.1
29.9
41.8
38.5
38.4
Grayscale Reverse Mask
45.2
40.4
40.5
43.8
50.9
45.8
45.9
51.2
Blur Reverse Mask
67.8
43.3
52.8
58.0
63.5
52.8
55.4
57.8
Table 2: Ablation study the zero-shot performance of individual visual prompting in the validation of COCO, PACO, RefCOCO, RefCOCO+, and RefCOCOg datasets using ground truth annotations (left) and proposals in referring expression comprehension (right), respectively. Best metrics are in bold, and sub-optimal results are underlined.
instances. The COCO dataset is annotated with boxes and masks for objects, while PACO additionally incorporates annotations of corresponding parts for each object.
4.2 Implementation Details
We follow Timm [57], CPT [61], and ReCLIP [50] construct our pipelines. By default, we use ViT-B/32, ViT-L/14@ 336px, and RN50x16 backbones trained in CLIP [42] by OpenAl as our vision- language model. To simplify reference to these models in subsequent experiments, we refer to them as ViT-B, VIT-L, and RN. Additionally, for improved mask generalization, we employ SAM-Vij VIT-H, a variant Segment Anything Model [26]. Concerning part detection on the PACO [44] dataset, we begin by cropping the object according to the annotated boxes and then perform part detection on each object crop. All experiments are conducted on 8x Tesla V100.
4.3 Ablation Study on Visual Prompt
In this section, we elaborate on the zero-shot per- formance of individual visual prompts defined in Fig. 1 with two experiment settings. Firstly, we assume prompting representations, i.e.,
Blur Reverse Mask
65-
bounding boxes and semantic masks, to be as precise as possible, in order explore the up- per bound of each visual prompt. Therefore, we acquire them from ground truth annotations of all datasets. Then we evaluate the image- label matching accuracy following the zero-shot pipeline described in Sec. 3.1. More specifically, instance-label pairs are represented as object- object name, part-part name, and object-caption in COCO, PACO, and RefCOCO series datasets, respectively. The left part of Table 2 reveals that the Blur Reverse Mask [D4] offers the best overall performance.
Accuracy
55
50
RdCOCO md
RcROCO
Re/oOCO
Ra00CO JohA
Rc20C0 JaLA
Re/00CO
40
RacoCo
10
20
50
100
200
500
1000
Standard Deviation
Figure 5: Ablation the standard deviation in Gaussian blur kernel from the Blur Reverse Mask [D4] prompting. larger deviation presents a more blurred background.
In the second part, we ablate on the referring expression comprehension task under the codebase of ReCLIP [50] with proposal boxes provided by MAttNet [62]. Note that in this setting, semantic

---

RefCOCO
RefCOCO+
RefCOCOg
Method
Backbone
Visual Prompt
PP
val
lestA
testB
val
lestA
testB
val
test
Our codebase with object proposals detected UNINEXT
[60]
Crop
VIT-I
P
45.3
47.2
43.7
46.4
48.7
42.9
56.4
56.3
RedCirele [49]
VIT-L
C1
48.9
56.4
41.3
51.7
59.5
41.8
54.6
54.7
FGVP (ours)
VIT-I
D44
52.8
56.6
46.4
55.4
62.0
46.6
57.8
58.3
CPT [67] codebuse with object proposals detected by MAnNet
[62]
CPT [61]
VinVI. [66]
A2
32.2
36.1
30.3
31.9
35.2
28.8
36.7
36.5
RedCircle 1
ViT-L
C1
38.0
45.3
32.9
43.9
51.0
37.1
47.2
47.3
PGVP (ours)
VIT-I
D)4
46.1
53.0
40.4
50.4
57.5
42.6
54.5
54.1
ReCLIP [50] codebase with object proposals detected by MArtNer [62]
CPT-adapted [
VIT-B, RN
H2
R
23.2
21.4
27.0
23.9
21.6
25.9
22,3
23.7
CPT-adapted b
VIT-B, RN
P B2
R
41.3
40.6
44.0
41.3
41.8
41.1
51.3
51.2
ReCLIP [50]
VIT-B, RN
PI B4
R
45.8
46.1
47.1
47.9
50.1
45.1
59.3
59.0
RedCircle [49]
VIT-B, RN
PIC1
R
43.9
46.2
44.1
45.3
47.9
43.1
57.3
56.3
FGVP (ours)
VTT-B, RN
P D4
R
52.0
55.9
48.8
53.3
60.4
46.7
62.1
61.9
RedCircle [49]
VIT-L, C C4
49.8
58.6
39.9
63.9
45.4
59.4
58.9
RedCircle [49]
VIT-I, RN
C1 | C3 C4
S
51.4
58.3
40.9
56.3
63.6
45.8
58.3
58.0
FGVP (ours)
VIT-L, RN
D1 D3 D4
S
52,9
59.6
43.9
57.4
64.8
46.3
58.1
58.3
RedCircle [49]
VIT-L, RN
P C1
C3 C4
S
51.6
58.0
42.0
58.1
64.5
47.5
60.0
59.3
FGVP (our)
VIT-L, RN
P D1
D3
DA
S
53.9
60.2
44.3
59.3
66.6
48.8
61.0
61.3
RedCirele [
VIT-I,,
C3
C74
RS
56.8
62.4
49.1
58.6
64.7
48.3
62.2
61.0
FGVP (ours)
VIT-L,
D3
D4
59.6
52.0
66.8
63.3
Table 3: The performance of the rec benchmarked with ReSCOCO, RefCOCO+, and ReiCOCOg datasets. The Visual Prompting strategics shown in Fig. 1. PP: Post Processing. "R" and "S" denote Spatial Relations [50] and Score Subtraction [49], respectively. FGVP: Fine-Grained Visual Prompting. :CPT-adapted is an adapted version CPT-BIk implemented by ReCLIP. + Re-implementation performance. The best result for each dataset, w.r.t. each codebase is in bold.
masks in FGVP are derived following our proposed framework presented Fig. 2. Consequently, the Blur Reverse Mask [D4] shows a consistent superiority demonstrated in the right part of Table 2. Notably, lised [*1] prompting, we employ a default red line with a thickness of pixels, as described in the RedCircle [49]. As for color-based methods [+2], we utilize a green color with transparency level of 0.5 following CPT [61]. Next, we ablate the standard deviation of the Gaussian blur kernel for blur-based prompting [+4] (Fig. 5), and a value of 100 achieves the best. More ablation
study details on the prompting properties are provided in the Supplementary Materials.
4.4 Referring Expression Comprehension
In this section, we compare our FGVP with previous zero-shot methods in Table 3. For fair com- parisons, we also implement FGVP upon the original codebases of each compared method, namely CPT [61] and ReCLIP [50].
We conduct experiments following consistent settings with each compared work. To be specific, with the CPT and our codebase, we focus on individual visual prompting performance without post-processing. We use proposals from UNINEXT and MAttNet demonstrate the robustness of our enhancements. It's important to note that different proposal selections solely affect the box candidates, which are equitably shared among all the comparison prompting methods. Next, with the ReClip codebase, ReClip employs cropping and colorful boxes visual prompts, with default apatially-relation post-processing. To ensure a fair comparison, we first add cropping as an ensemble to all experiments. Next, to facilitate comparison with RedCircle (which inherently uses Score Subtraction as post-processing and primarily ensembles based on three circle prompts), we adopt the same three types of prompt formats but based on semantic masks. Finally, we explore higher performance possibilities. The preprocessing procedure for text inputs, including the prefix and clear-text principle, remains consistent across all codebases. Additionally, compare zero-shot

---

Visual Prompt
PACO
RefCOCC
ReICOCO+
Re(COCOg
NMS dx -11
NMS dx
Crop
34.3
20
NAS de +855
Accuracs
Keypoint
11.9
16.7
18.5
19.7
18
Circle
17.4
24.9
29.8
32.4
Colorful Mask
15.2
24.1
21.4
18.6
16
Blur Reverse Mask
18.3
44.9
49.6
32
Grid Size
Table 4: Accuracy of the part detection with ViT-I. the validation set of each benchmark. The best result bold.
Figure 6: Ablation study the NMS threshold and grid size.
Image
Cri-WWise Keypoints
Proposals (Sample)
Prompting (Sample)
Prediction
Ground Truth
Figure 7: Visualization of the candidate grid-wise keypoints, the proposals, the visual prompting image, and the predicted results part detection.
referring accuracy under unified CLIP models. Unless otherwise stated, all existing method results reported according their original papers.
Table shows that the single fine-grained visual prompt, i.e., Blur Reverse Mask can surpass previous works. With a VIT-L model, our FGVP surpasses RedCircle by an average of 3.4% accuracy in our codebase and 5.5%~~8. % with the CPT [61] codebase. Further, we implement FGVP under the ReCLIP [50] codebase, which investigates the ensemble of multiple models, prompts, and post ensemble
processing. The results show that the improvement brought by FGVP orthogonal techniques. Under equal comparisons, our FGVP shows better performance than previous works with a consistent gain of an average of ~3% and a max of 12.5% in the RefCOCO+ testA set.
Part Detection
The PACO [44] dataset features annotations for boxes and masks for common objects and their corresponding parts. For the part detection task, models need to locate the part within object. Different from the referring task, there is prior information indicating where the parts are. Our proposed pipeline (Fig. 3) is capable of performing zero-shot part detection without box proposals and utilizing only image inputs in these circumstances. Different from the localization keypoints operation employed in RedCircle [40] which only predicts the center location, we instead predict precise semantic masks of the target. Same with the metric in referring expression comprehension, predicted box is considered correct only when the intersection over union with the ground truth box exceeds 0.5. As shown in Table our proposed method, FGVP, outperforms other coarse prompts for part detection. Notably, we set the grid size to 16 along one side of the image and used an NMS threshold of 0.7 by default. However, better performance proposals through the use of larger grid size and NMS threshold. We conduct an ablation study on them by varying one while fixing the other, illustrated in Fig. 6. visualization of the FGVP results is depicted in Fig.
be achieved by including more
In addition, the referring expression comprehension task can be instance can be seen as a part the image) when no prior box proposals are provided. We conducted zero-shot experiments utilizing only images as inputs. As shown in Table 4, the Blur Reverse Mask Prompting without prior box proposals even surpasses certain coarse visual prompting methods with box proposals.
part detection (cach referring
the validation set of RefCOCO, RefCOCO+, and RefCOCOg datasets,
4.6 Limitations
Firstly, FGVP takes longer for inference than other methods because it involves the segmentor to produce semantic masks. To be specific, we compare the inference costs in terms of computation and

---

Method
SAM scale
Mask-filter
CUDA Memory
Inference Time
IPS
Acc
Crop
0.91 GB
4.49 min
5.03
45.3
RedCircle [49]
0.91 GB
4.00 min
5.64
48.9
X
1.32
5.20 min
51.7
base
/
1.32 GB
27.47 min
0.82
52.1
x
2.14 GB
6.29 min
3.59
51.0
FGVP (ours)
large
/
2.14 GB
27.49 min
0.82
52.2
x
3,42 GB
7.34 min
3.08
51.9
huge
/
3,42 GB
28.02 min
0.81
52.8
Table 5: Comparisons of inference cost. IPS: Image per GPU second.
speed between our method and others. Further, we conduct an ablation study on the scalability of SAM. Notably, the post-processing technique filter small disconnected regions and holes in masks can further improve performance at the cost of speed. Disabling the mask-filter post-processing will greatly improve the speed without losing too much performance (Table 5), Experiments run on RefCOCO with a CLIP pre-trained ViT-L/14G0 336px on 8x NVIDIA A100, Above all, speeding up the architecture is an important direction for future improvement.
Secondly, the current implementation does not couple the visual prompt with a specifically designed language prompt. Therefore, the image-text alignment comprehension ability of VLM has been fully explored appropriately. Another consideration is that there are more possible tasks that can adopt the proposed method for zero-shot transfer, such referring image segmentation. We leave it to our future work.
Conclusion
In this paper, focus the visual prompt, a technique to highlight the target instances in the image content using visible markings such as colorful boxes and circles. The visual prompt is beneficial in invoking potential spatial comprehension within VLMs like CLIP instance-level tasks. However, existing prompting designs are often too coarse for locating the target instance, leading to unrelated information that may harm performance. Since the topic of visual prompting is rarely explored, we systematically summarize various typical prompting formats and propose Fine-Grained Visual Prompting (FGVP). which utilizes precise semantic masks of target instances derived via Segment Anything (SAM). We discover that the Blur Reverse Mask prompting, which blurs the background, achieves the best performance. Furthermore, we construct two zero-shot classification architectures for regular referring expression comprehension tasks using boxes as prior proposals and for part detection utilizing only input images. We evaluate the effectiveness of FGVP on referring expression comprehension and achieve state-of-the-art (SOTA) performance.
Acknowledgement
This project is supported by the National Key R&D Program of China (2022D01163002 the Young Scientists Fund of the National Natural Science Foundation of China (Grant No. 62206134), 13)), the Fundamental Research Funds for the Central Universities (070-63233084), and the Tianjin Key Laboratory of Visual Computing and Intelligent Perception (VCIP). Computation is partially supported by the Supercomputing Center Nankai University (NKSC). Note that Lingfeng Yang and Jian Yang are with Jiangsu Key Lab of Image and Video Understanding for Social Security, and Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, Nanjing University of Science and Technology, Nanjing, 210094, P.R. China, while Lingfeng Yang and Xiang Li are with IMPlus PCALab.
Broader Impacts
Further research and careful consideration are necessary when utilizing this technology, as the presented proposed method relies on statisties derived from training datasets that may possess biases and could potentially result in negative socictal impacts.
10

---

References
[1] Jean-Baptiste Alayrae, Jeff Donahue, Pauline Lue, Antoine Micch, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurlPS, 2022.
[2] Hyojin Bahng, Ali Jahanian, Swami Sankaranarayanan, and Phillip Isola. Exploring visual prompts for adapting large-scale models. arXiv preprint ar:v::2033. 172744 2022.
[3] Hanoona Bangalath, Muhammad Maaz, Muhammad Uzair Khattak, Salman H Khan, and Fahad Shah- baz Khan. Bridging the gap between object and image-level representations for open-vocabulary detection. NewrIPS, 2022.
[4] Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei Efros. Visual prompting via image inpainting. NeurlPS, 2022.
[5] Omer Bar-Tal Dolev Ofri-Amat Rafail Fridman. Yoni Kasten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In ECCV, 2022.
[6] Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. arXiv preprint arXiv:2211.0995 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Ncelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot leamers.. NewrIPS, 2020.
[8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[9] Xianjie Chen, Roozbeh Mouaaghi Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. In CVPR, 2014.
[10] Yen-Chun Chen, Linjie Li. Licheng Yu, Ahmed El Kholy, Faisal Ahmed. Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020.
[11] Bowen Cheng, Alex Schwing, and Alexander Kirillov, Per-pixel classification is not all you need for semantic segmentation. NeurlPS, 2021.
[12] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.0233 2022.
[13] Jifeng Dai, Kaiming He, and Jian Sun. Instance-aware semantic segmentation via multi-task network cascades. In CVPR, 2016.
[14] Du Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Learning to prompt for open-vocabulary object detection with vision-language model. In CVPR, 2022.
[15] Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. CVPR, 2023.
[16] Chengjian Feng, Yujie Zhong, Zequn Jic, Xiangxiang Chu, Haibing Ren, Xiaolin Wei, Weidi Xie, and Lin Ma. Promptdet: Towards voca-vocyy detection using uncurated images. ECCV, 2022.
[17] Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. CVPR, 2014.
[18] Ke Gong, Xiaodan Liang, Yicheng Li. Yimin Chen, Ming Yang, and Liang Lin. Instance-level human parsing via part grouping network. In ECCV, 2018.
[19] Ke Gong, Xiaodan Liang, Dongyu Zhang, Xiaohui Shen, and Liang Lin. Look into person: Self- Self-supervised cructure-sensitiie learning and new benchmark human parsing. In CVPR, 2017.
[20] Xiuye Gu, Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open- Open-vocabulary object detection via vision and language knowledge distillation. ICLR, 2022.
[21] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In ICCV, 2017.
[22] Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Nem Lim. Visual prompt tuning. In ECCV, 2022.

---

[23] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modullted detection for end-to-end multi-modal understanding. In ICCV, 2021.
[24] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Aszion-and-lngguag transformer without convolution or region supervision. In ICMI,, 2021.
[25] Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmentation. In CVPR, 2019.
[26] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. arXiv preprint arXiv:2304.0260 2023.
[27] Harold W Kuhn. The hungarian method for the assignment problem. Naval research logistics quarterly. 1955.
[28] Christoph H Lampert, Hannes Nickisch, and Stefan Harmeling. Learning to detect unseen object classes by between-class attribute transfer. In CVPR, 2009.
[29] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2lip-2: Bootstrapping language- annuage-imaae pre- pre-training with frozen image encoders and large language models. arXiv preprint (..iv:2331.125999 2023.
[30] Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-imag pre- pre-training for unified vision-language understanding and generation. In ICML, 2022.
[31] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NeurlPS, 2021.
[32] Liunian Harold Li. Pengchuan Zhang. Haotian Zhang. Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Yuan, Lei Zhang. geng-Neng Hwang, Grounded language- image pre- pre-training. CVPR. 2022.
[33] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Oscar: Object-semantics aligned pre-training for rision-language tasks. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 23-28, 28, 2020, Proceedings,
Part XXX 16, pages 121 137. Springer, 2020.
[34] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language-image pre-training via masking. arXiv preprint arXiv:2212.0079) 2022.
[35] Feng Liang. Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. arXiv preprint ar Xiv: 10.04150, 2022.
[36] Tsung- Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[37] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, arXiv preprint arXiv:2303.05400 2023.
[38] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In CVPR, 2015.
[39] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan I. Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016.
[40] Rafael Müller, Simon Korblithh, and Geoffrey Hinton. When does label smoothing help? arXiv preprint MXXiv:906.006622 2019.
[41] OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.0877 2023.
[42] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In ICMI. 2021.
[43] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to to-text transformer. The Journal of Machine Learning Research, 2020.
12

---

[44] Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang. Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. arXiv preprint 0iii::2301.011955, 2023.
[45] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.01125, 2022.
[46] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
[47] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
[48] Taylor Shin, Yasaman Razeghi, Robert L Logan IV. Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010. 15980. 2020.
[49] Aleksandar Shtedritski, Christian Rupprecht, and Andrea Vedaldi. What does clip know about a red circle? visual prompt engineering for vims. arXiv preprint arXiv:2304.06712 2023.
[50] Sanjay Subramanian, Will Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. Re- clip: A strong zero-shot baseline for referring expression comprehension. arXiv preprint 2204:2204.00991, 2022.
[51] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 15389, 2023.
[52] Richard Szeliski. Computer vision: algorithms and applications. Springer Nature, 2022.
[53] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.1399 2023.
[54] Catherine Wah, Steve Branson, Peter Welinder, Pictro Perona, and Serge Belongic. The caltech-uesd 20rds-200-2011 dataset. Califormia Institute of Technology, 2011.
[55] Xinlong Wang, Tao Kong, Chunhua Shen, Yuning Jiang, and Lei Li. Solo: Segmenting objects by locations. In ECCV, 2020.
[56] Xinlong Wang, Xiaosong Zhang, Yue Cao, Wen Wang, Chunhua Shen, and Tiejun Huang. Seggpt: Segmenting everything context. arXiv preprint arXiv:2304.03288 032844 2023.
[57] Ross Wightman. Pytorch image models. httpp:///gituub..mm///ggggge 2019.
[58] Fangting Xia, Peng Wang. Liang-Chieh Chen, and Alan L Yuille. Zoom better to see clearer: Human and object parsing with hierarchical auto-zoom net. In ECCV, 2016.
[59] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai. A simple baseline for open- vocabulayy semantic segmentation with pre- pe-ttrineed vision- language model. In ECCV, 2022.
[60] Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception as object discovery and retricval. arXiv preprint arXiv:2303.0661 2023.
[61] Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, and Maosong Sun. Cpt: Colorful prompt tuning for pre-trained lanion-language models. arXiv preprint arXiv:2 11797, 2021.
[62] Licheng Yu, Zhe Lin, Xiaohui Shen, Jimei Yang. Xin Lu, Mohit Bansal, and Tamara L Berg. Mattnet: Modular attention network for referring expression comprehension. CVPR, 2018
[63] Licheng Yu, Patrick Pairson, Shan Yang, Alexander Berg, and Tamara I. Berg. Modeling context in referring expressions. In ECCV, 2016.
[64] Yuhang Zang, Wei Li. Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In ECCV, 2022.
[65] Ning Zhang, Jeff Donahue, Ross Girshick, and Trevor Darrell, based r-cans for tin-ggrined category detection. In ECCV, 2014.
13

---

[66] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang, Lijuan Wang. Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In CVPR, 2021.
[67] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
[68] Shiyu Zhao, Zhixing Zhang, Samuel Schulter, Long Zhao, BG Vijay Kumar, Anastasis Stathopoulos. Manmohan Chandraker, and Dimitris N Metaxas. Exploiting unlabeled data with vision and language models for object detection. ECCV, 2022.
[69] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning prompt for vision-language models. 1/CV, 2022.
14


Reviews:
[{"review_id": "kGLRBtHCFp", "reviewer": "Reviewer_Xx88", "summary": "The paper proposes Fine-Grained Visual Prompting (FGVP), which uses precise semantic masks from SAM as visual prompts to improve spatial localization of vision-language models (VLMs) like CLIP for instance-level tasks. The key contributions are:\n\n- Systematically study different visual prompting techniques like cropping, boxes, circles, masks, etc. Show that blurring background outside target mask (Blur Reverse Mask) works best.\n- Achieve state-of-the-art results on referring expression comprehension benchmarks RefCOCO/RefCOCO+/RefCOCOg, outperforming prior works.\n- Demonstrate FGVP can enable zero-shot part detection on PACO dataset without needing any box proposals, again outperforming other prompting methods.", "questions": "See the main suggestions in weakness. \n\n- For the RegCOCO/RefCOCO+/RefCOCOg results in Table 4, which subsets are the numbers reported on?\n- Line 288-290 if there are better grid size / NMS thresholds, how do we still use the suboptimal default ones?", "limitations": "Limitations are discussed in the paper", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Originality: The paper presents a novel idea that using SAM to generate semantic masks from detected bbox as better visual prompts of specific instance in the image, which is an original idea not explored in prior works.\n\nQuality: The overall approach is technically sound. The experiments follow standard protocols and are extensive in studying different visual prompting design. The results demonstrate benefits over existing methods.\n\nClarity: The paper is well-written and easy to follow. The problem context, proposed method, experiments are clearly explained. Figures and tables aid understanding.\n\nSignificance: FGVP pushes state-of-the-art in two important vision-language tasks - referring expression and part detection. The analysis may inspires more research on the properties of VLMs regarding spatial understanding.", "weaknesses": "While the paper presents a novel fine-grained visual prompting technique and achieves state-of-the-art results, there are some aspects where the analysis could be strengthened:\n\nMore Insights from Study: The paper performs an extensive set of experiments on different visual prompt designs. However, it could provide a more detailed analysis of the inferences and insights derived from this study. For instance, comparing the gap between using ground truth and predicted bboxs would give insights into the impact of mask quality. Explaining the differences in various design choices (VP, PP, proposals etc.) in Table 3 would be informative. Attention visualizations could help reveal why fine-grained prompting is more beneficial.\n\nComputational Overhead: The paper proposes generating semantic masks using SAM models. However, the computational overhead this introduces is not analyzed. Reporting inference times, scalability, etc. would provide a better understanding of its practical viability.\n\nJoint Language and Visual Prompting: The study is currently limited to exploring only visual prompts. Evaluating prompts on both visual and textual modalities could offer a more comprehensive understanding of VLMs. It is good that authors discussed this point in the limitation though.\n\nGranular and Failure Analysis: Providing per-class performance breakdowns and detailed failure analysis compared to other methods through examples would provide useful insights into where the improvements come from.\n\nIn summary, while the core ideas are promising, performing a more thorough empirical analysis along the above dimensions would strengthen the paper and provide a better understanding of the factors behind the efficacy of fine-grained visual prompting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments and suggestions!\n\n**Q1**: More insights from the study.\\\n**A1**: \\\n**1)** Compare the gap in mask quality caused by using ground truth and predicted boxes.\\\n**Firstly**, Table 2 in the main paper contrasts results with masks derived from ground truth (left side) and proposal boxes (right side).\\\n**Secondly**, we broadened experiments by employing SAM at varied scales and an unsupervised mask generator from FreeSOLO, yielding diverse mask qualities. We present the performance of FGVP with the Blur Reverse Mask prompting. Generally, FGVP achieves the highest accuracy with ground truth masks, reinforcing the rationale for \"Using Ground Truth as Performance Upper Bound.\" Larger SAM backbones typically yield better masks, resulting in higher performance, while the unsupervised mask generator results in the lowest performance for poor mask qualities.\n\n|Mask source|SAM scale|COCO|PACO|RefCOCO|RefCOCO+|RefCOCOg|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|GT mask|\\--|67.8|42.7|52.8|58.0|63.5|\n|GT box|huge|68.0|39.5|52.3|56.1|62.2|\n|GT box|large|67.9|39.8|51.6|55.6|61.3|\n|GT box|base|67.4|39.1|52.1|55.2|60.8|\n|UNINEXT proposal|huge|\\--|\\--|52.8|55.4|57.8|\n|UNINEXT proposal|large|\\--|\\--|52.2|54.8|57.5|\n|UNINEXT proposal|base|\\--|\\--|52.1|54.5|58.8|\n|Mask generator of FreeSOLO|\\--|17.4|11.6|27.7|30.5|38.2|\n\n\n**2)** Design choices with the visual prompt (VP), post-processing (PP), proposals, etc. in Table 3.\\\n**Above all**, the guiding principle for all settings is to maintain consistency with the compared works.  \n**To be specific**: \n- **CPT and our codebase** focus on individual VP performance without PP. We use UNINEXT and MAttNet proposal banks to demonstrate the robustness of our enhancements. It's important to note that different proposal selections solely affect the box candidates, which are equitably shared among all the comparison prompting methods.\n- **For the ReClip codebase**, the ReClip employs cropping and colorful boxes as visual prompts, with default spatially-relations post-processing. To ensure a fair comparison, we **first** add cropping as an ensembled VP to all experiments. **Next**, to facilitate comparison with RedCircle (which inherently uses Score Subtraction as post-processing and primarily ensembles based on three circle VPs, as summarized in Table 1), we adopt the same three types of prompt formats but based on semantic masks. **Finally**, we aim to explore a higher performance possible under various VP and PP ensembles.\n\n\n**3)** Attention visualizations between fine-grained prompting and previous methods.\\\nFollowing the pipeline established in DINO, we present attention visualizations in **Figure R3 of the PDF in the global response**. We compare Crop, RedCircle, and FGVP using the RefCOCO datasets. The green number in the top left corner denotes the similarity score and is arranged in descending order. Each correct prediction is indicated by a red rectangle.\\\nThe visualization results show the reasons for the superior performance of FGVP and the limitations of RedCircle. This is because FGVP demonstrates a more reasonable behavior in terms of reducing attention to weakly correlated or distracting backgrounds and enhancing focus on the target object, especially small objects. While RedCircle can also alter attention allocation, the degree is quite limited, resulting in inferior performance. \\\nAt the same time, the attention analysis effectively validates the main perspective of our paper regarding the capability of the reverse blur mask to reduce attention on weakly correlated pixels and enhance focus on the main subject. Therefore, the proposed FGVP holds better potential in the field.\n\n**Q2**: Experiments on the computational overhead, inference times, and scalability.\\\n**A2**: Please refer to the **global response A1 and A2** for detailed information.\n\n**Q3**: Joint Language and Visual Prompting.\\\n**A3**: We appreciate your insightful comment and constructive guidance and have included two experiments for better understanding.\\\n**1)** **Firstly**, there is a general approach to text prompts that employs SPACY to extract nouns from captions. For instance, given a caption like \"a dog on the table,\" the noun extracted would be \"dog.\" Subsequently, the similarity score $S$ can be derived through an ensemble of the caption score $S_c$ and the prompted noun score $S_n$ : $S = r \\times S_c + (1 - r) \\times S_n$, where $r$ represents the balance ratio.\n\n||caption ($r$=1)|noun ($r$=0)|ensemble ($r$=0.5)|\n|:-:|:-:|:-:|:-:|\n|example|a dog on the table|dog|\\--|\n|FGVP|52.8|53.1|54.4|\n\n**2)** **Furthermore**, we have explored strategies for rephrasing text based on visual prompts. For instance, when presented with an image prompt featuring a blurred background, the original text \"a dog on the table\" could be rephrased as \"a dog on the table with a blurred background\".\n\n||caption|<caption> with blurred background|\n|:-:|:-:|:-:|\n|example|a dog on the table|a dog on the table **with blurred background**|\n|FGVP|52.8|54.8|\n\n**Q4**: Per-class performance and comprehensive failure analysis compared to other methods.\\\n**A4**: Please refer to the **global response A3**.\n\n**Q5**: Other issues.\\\n**A5**: \\\n**1)** The performance reported in Table 4 is derived from the validation set of all datasets.\\\n**2)** Using the suboptimal settings in Table 4 while there are better grid size / NMS Thresholds.\\\nThe grid size and NMS threshold we employed are default settings used in the official codebase of SAM. We avoided a higher setting due to increased inference cost and lengthy times, as discussed in **global response A2**. \\\n**Regarding your query**, an ablation study with larger settings (NMS threshold = 0.9, grid size = 32) is included, showing the continued superiority of our FGVP.\n\n||PACO|RefCOCO|RefCOCO+|RefCOCOg|\n|:-:|:-:|:-:|:-:|:-:|\n|Crop|19.5|17.4|22.1|35.3|\n|RedCircle|19.9|25.9|31.0|35.0|\n|FGVP|23.2|42.1|46.0|50.7|"}, {"review_id": "ckSe4htn3J", "reviewer": "Reviewer_jMVP", "summary": "This paper proposes a visual prompting method that exploits the segmentation masks of interested objects in images to generate more fine-grained visual prompts. Experiments show that the proposed methods achieve competitive results on zero-shot referring expressions comprehension and part detection.", "questions": "1. Basically, I like the section 3.3 of the discussion of the prior knowledge introduced by the large amount of photography images in the large-scale image-text dataset. Could the authors provide more discussions on the analysis of the training data of the VLMs, and the effect of better alignments between the visual prompt design and the VLM training data?  \n2. How good are the zero-shot results compared with the few-shot / many-shot methods? The few-shot / many-shot methods may be provided for general reference.", "limitations": "The proposed visual prompts can be further verified on more object-based tasks.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well-organized. The motivation and the framework is clearly presented.  \n2. The experiments confirm the effectiveness of the visual prompt design.", "weaknesses": "1. The ablation studies of different VLMs are missing. Since visual prompting is a zero-shot framework, it is natural that the performance of the visual prompts differ on VLMs trained on different data. The paper adopts the CLIP as the VLM in all experiments. How do different visual prompts perform on other VLMs?  \n2. Figures 2 & 3 are very similar and thus redundant. The authors should consider merging them into one figure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments and suggestions!\n\n**Q1**: Ablation studies on different VLMs. \\\n**A1**: The results across various VLMs demonstrate the consistent improvement of FGVP. Importantly, we observe that RedCircle experiences a significant performance decline when transitioning from CLIP to other models like SLIP, aligning with the results reported in the original RedCircle paper. In contrast, FGVP maintains its performance gain across different architectures.\n\n|Method|Backbone|Data|Params|Input size|Crop|RedCircle|FGVP|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|OpenAI CLIP|ViT-L/14@336px|CLIP-400M|304M|336|45.3|48.9|52.8|\n|OpenAI CLIP|ViT-B-32|CLIP-400M|87M|224|47.8|44.0|51.2|\n|Open CLIP|ViT-L/14|LAION-2B|304M|224|49.9|48.1|49.8|\n|SLIP|ViT-L/16|YFCC-15M|303M|224|44.3|33.7|49.3|\n|BLIP-v2|ViT-L/14|Merged-129M|304M|224|46.9|37.7|51.0|\n|EVA-02-CLIP|ViT-L/14@336px|Merged-2B|304M|336|51.4|51.5|55.8|\n\n**Q2**: Figures 2 & 3 are very similar and thus redundant. The authors should consider merging them into one figure.\\\n**A2**: Thank you for your proposal. There are significant design differences between the two mentioned frameworks. One requires a detector, while the other does not. The masks in the first framework are sparse, tied to proposal boxes, whereas the latter generates dense masks from SAM prompted with grid points. Describing and showcasing the two frameworks separately will help highlight the differences in design and effect details. After careful consideration, we chose to keep both frameworks independent.\n\n**Q3**: More discussions on the training data of the VLMs, and the effect of better alignments between the data and visual prompt design.\\\n**A3**: Thanks for valuing our discussion. We hope the biases analysis could provide more insight into the visual prompting-training data alignment. In **Figure R2 in PDF in the global response**, we added criminal text to regular COCO images with people. The RedCircle tends to classify the person into criminal classes while FGVP does not. We assume that a red circle is unnatural and lacks realism compared to web images. On the contrary, the image with a blurred background tends to look more natural and common rather than artificially post-processed, which helps to reduce unexpected bias. For detailed quantitative results, please refer to **Response to Reviewer Rf2R A2**.\n\n**In an ideal scenario**, our prompt may be better aligned with VLM training data by factoring in image depth. This involves gradual depth-based blurring incorporated with depth estimation results. Capturing natural shallow depth-of-field could be better than the current uniform blur.\n\n**Q4**: Compared with the few-shot / many-shot methods.\\\n**A4**: We present the best performance as reported in the original paper for the methods listed below. The results are partly summarized from the original papers of Pseudo-Q [1] and CPT [2].\\\n[1] Jiang H, Lin Y, Han D, et al. Pseudo-q: Generating pseudo language queries for visual grounding. CVPR, 2022.\\\n[2] Yao Y, Zhang A, Zhang Z, et al. Cpt: Colorful prompt tuning for pre-trained vision-language models. arXiv preprint arXiv:2109.11797, 2021.\n\n\n||||RefCOCO|RefCOCO|RefCOCO|RefCOCO+|RefCOCO+|RefCOCO+|RefCOCOg|RefCOCOg|\n|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|\n|Method|Published|Supervision|val|test-A|test-B|val|test-A|test-B|val|test|\n|MAttNet|CVPR’18|Full|76.7|81.1|70.0|65.3|71.6|56.0|66.6|67.3|\n|NMTree|ICCV’19|Full|76.4|81.2|70.1|66.5|72.0|57.5|65.9|66.4|\n|FAOA|ICCV’19|Full|72.5|74.4|68.5|56.8|60.2|49.6|61.3|60.4|\n|ReSC|ECCV’20|Full|77.6|80.5|72.3|63.6|68.4|56.8|67.3|67.2|\n|TransVG|ICCV’21|Full|80.3|82.7|78.1|63.5|68.2|55.6|67.7|67.4|\n|VC|CVPR’18|Weak|\\--|33.3|30.1|\\--|34.6|31.6|\\--|\\--|\n|ARN|ICCV’19|Weak|34.3|36.4|33.1|34.5|36.0|33.8|\\--|\\--|\n|KPRN|ACMMM’19|Weak|35.0|34.7|37.0|36.0|35.2|37.0|\\--|\\--|\n|DTWREG|TPAMI’21|Weak|39.2|41.1|37.7|39.2|40.1|38.1|\\--|\\--|\n|CPT|ArXiv’21|8-shot|41.3|48.2|35.7|42.6|49.3|35.4|47.4|47.4|\n|CPT|ArXiv’21|4-shot|40.7|47.4|35.3|40.3|46.5|34.5|44.4|44.4|\n|CPT|ArXiv’21|2-shot|39.8|45.6|33.9|38.6|44.5|32.8|44.7|44.3|\n|CPT|ArXiv’21|1-shot|37.2|41.5|33.2|37.9|42.3|33.9|43.1|43.4|\n|CPT|ArXiv’21|zero-shot|32.2|36.1|30.3|31.9|35.2|28.8|36.7|36.5|\n|Pseudo-Q|CVPR’22|zero-shot|56.0|58.3|54.1|38.9|45.1|32.1|46.3|47.4|\n|ReClip|ArXiv’22|zero-shot|45.8|46.1|47.1|47.9|50.1|45.1|59.3|59.0|\n|RedCircle|ArXiv’23|zero-shot|49.8|58.6|39.9|55.3|63.9|45.4|59.4|58.9|\n|FGVP (ours)|ArXiv’23|zero-shot|59.6|65.0|52.0|60.0|66.8|49.7|63.3|63.4|\n\n\n**Q5**: The proposed visual prompts can be further verified on more object-based tasks.\\\n**A5**: Thank you for your advice. We plan to extend FGVP to more grounding and open vocabulary benchmarks in our future works. Here, we provide experiments on two more object-based tasks.\\\n**1)** **Firstly**, we extended FGVP to open vocabulary detection based on the current state-of-the-art work OV-Seg [1].\\\n[1] Liang F, Wu B, Dai X, et al. Open-vocabulary semantic segmentation with mask-adapted clip. CVPR, 2023.\n|Method|Segmentor backbone|Clip|PAS-20|\n|:-:|:-:|:-:|:-:|\n|OVSeg|Swin-B|ViT-L|94.5|\n|OVSeg-Blur|Swin-B|ViT-L|95.1|\n\n**2)** **Next**, we experimented with FGVP on the Referring Image Segmentation benchmark. The table shows that we outperformed the current state-of-the-art Global-Local CLIP [2] by replacing feature cropping with our FGVP.\\\n[2] Yu S, Seo P H, Son J. Zero-shot Referring Image Segmentation with Global-Local Context Features. CVPR, 2023.\n\n|Method|Visual Encoder|oIoU|mIoU|\n|:-:|:-:|:-:|:-:|\n|Cropping|ViT-B/32|22.7|24.8|\n|Global-Local CLIP|ViT-B/32|24.8|26.2|\n|FGVP+Global-Local CLIP|ViT-B/32|25.1|26.6|\n"}, {"review_id": "0Qg6FKFkac", "reviewer": "Reviewer_Z75J", "summary": "This paper works on visual prompting. They proposed FGVP, together with Blur Reverse Mask, to improve the semantic localization ability of the vision-language model. ", "questions": "No further questions. ", "limitations": "No notable limitations.", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Solid experiments showing the effectiveness of their method.", "weaknesses": "In general, this is a good work. However, there’re several things concerning me:\n\n- The novelty of this work is not well established. Seems like an engineering combination of previous works.\n- The discussion of the upper bound is based on the assumption, while the legitimacy of the assumption is not well discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments and suggestions!\n\n**Q1**: The novelty of this work is not well established. Seems like an engineering combination of previous works.  \n**A1**: **Firstly**, thank you for your concern. With the development of large vision-language models and segmentors, they potentially embed knowledge for various downstream tasks. However, their basic usage mainly focuses on global-level tasks and class-agnostic segmentation. **They cannot be easily adopted to achieve high performance in instance-wise localization and classification tasks.** Besides, additional training for specific tasks end-to-end would be costly. Therefore, developing a zero-shot architecture is an elegant solution, which can unleash the potential of large models and combine prior knowledge to effectively tackle certain tasks without specific tuning. Simultaneously, such an architecture can also serve as a good baseline for further improvement, providing deeper insights for related work.  \n**Then**, we would like to reclaim our contributions and inspirations:  \n- To better align target regions with their captions, we propose to **highlight the target instances according to their semantic mask by blurring the background as fine-grained visual prompting**.  \n- After extensive experiments across diverse visual prompts, we demonstrate the superior performance of FGVP, which achieves SOTA performance on **zero-shot benchmarks**.  \n- The background blurring strategy is aligned with the natural photography images in web-scale data used for training VLMs, which **exploits potential knowledge within large models**. \n\n**In conclusion**, as mentioned by the reviewer Xx88, our work “**presents a novel idea that using SAM to generate semantic masks as better visual prompts, which is an original idea not explored in prior works**” and “**is a clever way to exploit 'natural supervision' present in large-scale web image training data**” by the reviewer Rf2R. For these reasons, we believe it is promising to further explore and improve the FGVP.\n\n\n\n**Q2**: The discussion of the upper bound is based on the assumption, while the legitimacy of the assumption is not well discussed.  \n**A2**: From the table in the response to **Reviewer Xx88 A1**, we conducted an ablation study of the performance under different mask qualities. The **ground truth masks achieve almost the best overall performance** compared to those generated by SAM or other segmentors. \\\n**Notably**, the masks generated by SAM achieve competitive or even better results compared to GT. An important reason is that the GT mask in the COCO dataset is annotated with relatively rough polygons, which has a certain gap with the high-quality mask from SAM.\\\n**In fact**, instead of considering the ground truth mask as an upper bound, it is mainly **served as a unified and constant evaluation benchmark to measure and compare different visual prompts**, as GT masks have already existed for convenient inference."}, {"review_id": "IvtMaA6043", "reviewer": "Reviewer_Rf2R", "summary": "This paper proposes a new “visual prompting” method. Visual prompting refers to the idea of altering images to guide the “attention” of a vision-language model when the model is used to embed the image. For example, to obtain an embedding for an object in an image that contains many objects, the user could draw a red circle around the object of interest and then embed the image. Since the vision-language model might have seen images during training in which important objects are highlighted with red circles, the resulting embedding might focus on the circled object.\n\nThis approach can be used to partially solve tasks such as referring expression comprehension: Given a set of bounding boxes and text descriptions, visual prompting can be used to find the best-fitting description for each box. A separate object detector is needed to obtain the bounding boxes first.\n\nThe paper proposes a visual prompting method that consists of blurring everything in the image except for the object of interest. This is done by first using a pretrained segmentation model (Segment Anything) to obtain a mask for the object of interest, and then blur everything outside of the mask. The motivation for this approach is that it simulates the shallow depth of field seen in photographs taken with a large aperture, which are commonly found in VLM training data.\n\nThe blur method is compared to the “red circle” method and several variants in object detection and referring expression tasks. The blur method consistently performs best. In addition, hyperparameter sweeps for the blur radius and other hyperparameters are shown.", "questions": "Please use descriptive method names (keypoint/box/circle/…) instead of just A/B/C in Table2 and elsewhere, so that the reader doesn't have to refer to Figure 1 all the time.", "limitations": "The runtime cost of the proposed method needs to discussed further.\n\nThe biases of the method compared to the \"red circle\" method need to be evaluated and discussed.\n\nA generic \"broader impact\" statement is given in the appendix.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed visual prompting method consistently improves over other methods.\n\n2. The blurring approach is well motivated by the abundance of photographs with shallow depth of field. This is a clever way to exploit “natural supervision” present in large-scale web image training data.\n", "weaknesses": "1. The proposed method is significantly more complex than the “red circle” method, since it relies on a large segmentation model. The inference cost of the proposed method is therefore much higher than the “red circle” method. This should be acknowledged in the discussion and/or limitation sections.\n\n2. While the proposed method works well, it is an incremental improvement over the “red circle” idea (https://arxiv.org/pdf/2304.06712.pdf) and the evaluation is not as comprehensive as in the “red circle” paper. For example, no analysis of the relative biases of red circle vs blur methods are performed, and no failure cases are discussed. It is not clear if the contribution is substantial enough for NeurIPS.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments and suggestions!\n\n**Q1**: The inference cost of the proposed method.\\\n**A1**: We present the inference cost comparison experiments in the **Global Response A1 and A2**. Thank you for your advice; we will acknowledge it in our limitation sections in the revised version.\n\n\n**Q2**: Analysis of the relative biases of red circle vs blur methods.\\\n**A2**: Following the analysis section in the paper of RedCircle, we **first** select one figure in COCO containing a male and a female. We visualize their categories classified by CLIP with added criminal-related texts under different visual prompting methods. The comparison of relative biases among RedCircle and FGVP is depicted in **Figure R2 in the PDF in the global response**. From the figure, RedCircle tends to classify persons into criminal categories, while FGVP and the original image without prompting manage to classify correctly. \nIt is mainly because a red circle is not a natural marking compared to the training web-scale data of CLIP. However, FGVP possesses more non-post-processed characteristics which helps to reduce the biases.\n\n**Next**, we quantify the biases following RedCircle based on the same datasets of FairFace and COCO. Additionally, we experimented with each visual prompting strategy on COCO with and without cropping the person out of the entire image as a pre-processing operation. From the following table, we can observe that although our FGVP might introduce a few more biases than the raw image, it substantially reduces biases compared to RedCircle due to its more natural prompting design.\n\n\n|Model|Visual Prompt|FairFace|COCO w/ crop|COCO w/o crop|\n|:-:|:-:|:-:|:-:|:-:|\n|ViT-L/14@336px|Crop|13.0|40.8|43.6|\n|ViT-L/14@336px|RedCircle|20.6 (+7.6)|49.9 (+36.9)|69.3 (+56.3)|\n|ViT-L/14@336px|FGVP|15.9 (+2.9)|34.1 (-6.7)|47.8 (+4.2)|\n|ViT-B/32|Crop|14.5|27.2|34.9|\n|ViT-B/32|RedCircle|22.0 (+7.5)|44.1 (+29.6)|68.6 (+54.1)|\n|ViT-B/32|FGVP|8.2 (-6.3)|19.5 (-7.7)|15.8 (-19.1)|\n|RN50×16|Crop|19.5|55.1|50.7|\n|RN50×16|RedCircle|38.5 (+19.0)|71.4 (+51.9)|72.6 (+53.1)|\n|RN50×16|FGVP|21.6 (+2.1)|56.0 (+0.9)|28.9 (-21.8)|\n\n\n**Q3**: Failure cases visualization and analysis. \\\n**A3**: Please refer to the failure case visualizations in **Figures R4 and R5 in the PDF document** and the corresponding analysis in the **global response A3**. **Lastly**, it's essential to highlight that we've established a comprehensive framework to facilitate evaluation comparisons among diverse visual prompting techniques and their post-processing ensembles. These aspects were not addressed by RedCircle.\n\n\n\n**Q4**: Use descriptive method names (keypoint/box/circle/…) instead of just A/B/C in the Table.\\\n**A4**: Thanks for pointing out this inconvenience. We will address this in the revised version.\n\n"}, {"review_id": "NHCp12AnF7", "reviewer": "Reviewer_TfGG", "summary": "This paper proposes Fine-Grained Visual Prompting (FGVP) that incorporates Blur Reverse Mask to improve the semantic localization capability of VLMs, like CLIP. It provides a comparison to other possible methods for highlighting the different parts/objects in the image, based on SAM and other techniques. The resulting method improves the performance on RefCOCO* datasets.  ", "questions": "Can SAM be replaced with a simpler segmentation model, like color-based unsupervised segmentation methods? \nAny ablation of SAM will be useful here for understanding the scale/quality that is required from the segmentation network for achieving good results. ", "limitations": "The limitation section is present and addresses some unexplored directions. I suggest adding runtime estimates.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is written in a clear way and describes the algorithm well. The evaluation shows clear improvements and the ablation study is convincing. The idea to blur and mask the background of different object/part proposals is both innovative and significant.   ", "weaknesses": "The related work section missing two relevant works in the visual prompting domain:\n\n1. Bhang et al., \"Exploring Visual Prompts for Adapting Large-Scale Models\", 2022\n\n2. Bar et al., \"Visual Prompting via Image Inpainting\", NeurIPS, 2022\n\n\nMoreover, while SAM is a powerful method for segmentation, it requires running the model with a relatively dense grid of keypoints. This runtime can be significant and should be discussed in the limitations section.  ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the comments and suggestions!\n\n**Q1**: Missing related works in the visual prompting domain.\\\n**A1**: Thanks, we will cite these works in the revised version.\n\n\n\n**Q2**: The runtime should be discussed since the method requires running the model with a relatively dense grid of keypoints.\\\n**A2**: Thanks for your advice, we will add and discuss the inference runtime limitations in the revised version.\\\n**Firstly**, we propose a method that uses the proposal boxes from a detector to generate masks. Consequently, proposals are sparse and their quantity depends on the outputs of the detector. In this situation, the time comparison results are presented in our **global response A1**.\\\n**Secondly**, under the framework where no detector is available and we need to use dense grid points as proposals, the runtime experiments are implemented in the **global response A2**.\n\n\n\n**Q3**: Can SAM be replaced with a simpler segmentation model, like color-based unsupervised segmentation methods? More ablation studies of SAM for understanding the scale/quality to achieve good results.\\\n**A3**: **First of all**, SAM could be replaced with another segmentor; we implement the unsupervised mask generator in FreeSOLO to produce masks. Also, we use SAM of different scales to understand how the scale/quality affects the final results. The results can be found in the **response to Reviewer Xx88 A1**. \\\n**Notably**, we also conducted more experiments in the supplementary materials (B.4 Robustness of Mask Precision) on the Robustness of Mask Precision by manually expanding or shrinking the mask derived from SAM. I hope it will provide more information about the mask quality."}]

Author rebuttal:
Thank you to all the reviewers for their valuable comments and suggestions! Here are the responses to some common concerns.

**Q1**: Experiments of inference cost with **detector proposals (Figure 2)**.\
**A1**: We conducted efficiency experiments, comparing inference costs in terms of computation and speed between our method and others. As FGVP relies on SAM for semantic masks, we ablate the **scalability** with various SAM backbone scales. Notably, the post-processing technique to filter small disconnected regions and holes in masks can further improve performance at the cost of speed. Disabling the mask-filter post-processing will **greatly improve the speed without losing too much performance**. Experiments are run on RefCOCO with a CLIP pretrained ViT-L/14@336px on 8×NVIDIA A100. Generally, FGVP takes more inference times than others, which will be acknowledged as a limitation and improvement direction in the revised version.

|Visual Prompt|SAM scale|Mask-filter|CUDA memory (GB)|Inference time (min)|Image per GPU second|Acc|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|Crop|\--|\--|0.91|4.49|5.03|45.3|
|RedCircle|\--|\--|0.91|4.00|5.64|48.9|
|FGVP|base|no|1.32|5.20|4.34|51.7|
|FGVP|base|yes|1.32|27.47|0.82|52.1|
|FGVP|large|no|2.14|6.29|3.59|51.0|
|FGVP|large|yes|2.14|27.49|0.82|52.2|
|FGVP|huge|no|3.42|7.34|3.08|51.9|
|FGVP|huge|yes|3.42|28.02|0.81|52.8|

**Q2**: Experiments of inference cost with **dense grid points as proposals (Figure 3)**.\
**A2**: Without provided detectors and proposals, we utilize SAM with dense grid points and an NMS threshold for mask filtering. We explore speed-performance trade-offs by varying grid sizes and NMS thresholds. All visual prompting methods, whether boxes, circles, or masks, rely on SAM for proposal yielding. Thus, the NMS threshold and grid size affect all methods equally. Experiments on PACO with a CLIP pretrained ViT-L/14@336px and SAM-huge on 8×NVIDIA A100 show that **FGVP could outperform RedCircle in speed and accuracy** at grid size 8 and NMS threshold 0.95 trade-off.

|Visual Prompt|Grid size|NMS threshold|Inference time (min)|Image per GPU second|Acc|
|:-:|:-:|:-:|:-:|:-:|:-:|
|Crop|16|0.7|13.34|3.27|16.5|
|Crop|32|0.95|37.25|1.17|19.5|
|RedCircle|16|0.7|12.75|3.42|17.4|
|RedCircle|32|0.95|34.18|1.28|19.9|
|FGVP|8|0.7|8.33|5.24|17.3|
|FGVP|8|0.95|9.17|4.76|20.5|
|FGVP|16|0.7|14.89|2.93|18.4|
|FGVP|16|0.95|17.29|2.52|22.0|
|FGVP|32|0.7|34.73|1.26|19.0|
|FGVP|32|0.95|39.66|1.10|23.2|

**Q3**: Granular and Failure Analysis\
**A3**: \
**1)** Per-class performance.\
We mainly compare the per-class performance of RedCircle and our FGVP on COCO. FGVP (Blur Reverse Mask) surpasses RedCircle in class accuracy for 53 out of 80 classes, particularly excelling in major categories like car, person, bird, and chair. However, FGVP is inferior to RedCircle in a few categories like the baseball glove and traffic light. We show all results in **Figure R1 in PDF in the global response**.

|Category|person|car|bird|chair|traffic light|tennis racket|
|:-:|:-:|:-:|:-:|:-:|:-:|:-:|
|Instance Number|10777|1918|427|1771|634|225|
|RedCircle|35.6|23.6|58.8|32.5|67.4|87.1|
|FGVP|76.2 (+40.6)|62.5 (+38.9)|91.6 (+32.8)|62.7 (+30.2)|52.8 (-14.6)|45.3 (-41.8)|

**2)** Detailed failure analysis compared with other methods.\
We visualize the challenging cases in **RefCOCO (Figure R4) and COCO (Figure R5) in the PDF file in the global response**.
- **For the RefCOCO dataset**, we find that all visual prompts performed poorly when the target object is difficult to be recognized due to its semantic gap or weak perceptual difference from the background. For example, in the sample Figure R4 (1), the "white shirt" mentioned in the caption is hardly visible in the image. In sample (2), FGVP grounds part of the "left bicycle" that is not cut off/blocked to the front wheel of the actual bicycle. At the same time, it is considered that the rest of the actual bicycle may belong to another bicycle. This is largely due to perceptual illusion. Samples (3) and (4) typically indicate that the performance bottleneck of FGVP could lie within SAM, yielding inaccurate masks and masks containing unrelated noises.
- **For the COCO dataset**, we observed that small objects can be accurately localized using positive masks rather than blurred masks. This indicates an improvement over the specific usage of semantic masks as FGVP for different types of instances.

**Consequently**, these difficult samples pose a general challenge to the algorithm. Overcoming such challenges becomes an important avenue for our future research. For instance, enhancing the involvement and connection of textual cues and exploring strategies like gathering and tuning visually difficult samples could be explored.

Meta-review: This paper has received overwhelmingly positive reviews with reviewers praising the originality, significance and clarity of the proposed approach in tackling the vision-language tasks of referring expression and part detection.
In addition to these strengths, reviewers also appreciated the strong empirical results on the mentioned tasks that demonstrate state-of-the-art performance on multiple datasets.
Finally, the rebuttals provided by the authors included additional experiments, ablation studies and observations that reviewers indicated as satisfactory in addressing some of the remaining questions raised in the initial reviews.
Decision: Accept (poster)