SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this section, we elaborate on the zero-shot performance of individual visual prompts defined in Fig. 1 with two experiment settings. Firstly, we assume all prompting representations, i.e., bounding boxes and semantic masks, to be as precise as possible, in order to explore the upper bound of each visual prompt. Therefore, we acquire them from ground truth annotations of all datasets. Then we evaluate the image-label matching accuracy following the zero-shot pipeline described in Sec. 3.1. More specifically, instance-label pairs are represented as object-object name, part-part name, and object-caption in COCO, PACO, and RefCOCO series datasets, respectively. The left part of Table 2 reveals that the Blur Reverse Mask  $[D4]$  offers the best overall performance.
Figure 5: Ablation on the standard deviation in Gaussian blur kernel from the Blur Reverse Mask [D4] prompting. A larger deviation presents a more blurred background.
In the second part, we ablate on the referring expression comprehension task under the codebase of ReCLIP [50] with proposal boxes provided by MAttNet [62]. Note that in this setting, semantic
Method	Backbone	Visual Prompt	PP	RefCOCO	RefCOCO+	RefCOCOg
val	testA	testB	val	testA	testB	val	test
Our codebase with object proposals detected by UNINEXT [60]
Crop	ViT-L	P	-	45.3	47.2	43.7	46.4	48.7	42.9	56.4	56.3
RedCircle [49] †	ViT-L	C1	-	48.9	56.4	41.3	51.7	59.5	41.8	54.6	54.7
FGVP (ours)	ViT-L	D4	-	52.8	56.6	46.4	55.4	62.0	46.6	57.8	58.3
CPT [61] codebase with object proposals detected by MAttNet [62]
CPT [61]	VinVL [66]	A2	-	32.2	36.1	30.3	31.9	35.2	28.8	36.7	36.5
RedCircle [49] †	ViT-L	C1	-	38.0	45.3	32.9	43.9	51.0	37.1	47.2	47.3
FGVP (ours)	ViT-L	D4	-	46.1	53.0	40.4	50.4	57.5	42.6	54.5	54.1
ReCLIP [50] codebase with object proposals detected by MAttNet [62]
CPT-adapted [50]‡	ViT-B, RN	B2	R	23.2	21.4	27.0	23.9	21.6	25.9	22.3	23.7
CPT-adapted [50]‡†	ViT-B, RN	P | B2	R	41.3	40.6	44.0	41.3	41.8	41.1	51.3	51.2
ReCLIP [50]	ViT-B, RN	P | B4	R	45.8	46.1	47.1	47.9	50.1	45.1	59.3	59.0
RedCircle [49] †	ViT-B, RN	P | C1	R	43.9	46.2	44.1	45.3	47.9	43.1	57.3	56.3
FGVP (ours)	ViT-B, RN	P | D4	R	52.0	55.9	48.8	53.3	60.4	46.7	62.1	61.9
RedCircle [49]	ViT-L, RN	C1 | C3 | C4	S	49.8	58.6	39.9	55.3	63.9	45.4	59.4	58.9
RedCircle [49] †	ViT-L, RN	C1 | C3 | C4	S	51.4	58.3	40.9	56.3	63.6	45.8	58.3	58.0
FGVP (ours)	ViT-L, RN	D1 | D3 | D4	S	52.9	59.6	43.9	57.4	64.8	46.3	58.1	58.3
RedCircle [49] †	ViT-L, RN	P | C1 | C3 | C4	S	51.6	58.0	42.0	58.1	64.5	47.5	60.0	59.3
FGVP (ours)	ViT-L, RN	P | D1 | D3 | D4	S	53.9	60.2	44.3	59.3	66.6	48.8	61.0	61.3
RedCircle [49] †	ViT-L, RN	P | C1 | C3 | C4	RS	56.8	62.4	49.1	58.6	64.7	48.3	62.2	61.0
FGVP (ours)	ViT-L, RN	P | D1 | D3 | D4	RS	59.6	65.0	52.0	60.0	66.8	49.7	63.3	63.4
Table 3: The performance of the rec benchmarked with RefCOCO, RefCOCO+, and RefCOCOg datasets. The Visual Prompting strategies are shown in Fig. 1. PP: Post Processing, "R" and "S" denote Spatial Relations [50] and Score Subtraction [49], respectively. FGVP: Fine-Grained Visual Prompting. tCPT-adapted is an adapted version of CPT-Blk implemented by ReCLIP.  $\dagger$  Re-implementation performance. The best result for each dataset, w.r.t. each codebase is in bold.
masks in FGVP are derived following our proposed framework presented in Fig. 2. Consequently, the Blur Reverse Mask  $[D4]$  shows a consistent superiority demonstrated in the right part of Table 2. Notably, in line-based  $[*1]$  prompting, we employ a default red line with a thickness of 2 pixels, as described in the RedCircle [49]. As for color-based methods  $[*2]$ , we utilize a green color with a transparency level of 0.5 following CPT [61]. Next, we ablate the standard deviation of the Gaussian blur kernel for blur-based prompting  $[*4]$  (Fig. 5), and a value of 100 achieves the best. More ablation study details on the prompting properties are provided in the Supplementary Materials.
4.4 Referring Expression Comprehension
In this section, we compare our FGVP with previous zero-shot methods in Table 3. For fair comparisons, we also implement FGVP upon the original codebases of each compared method, namely CPT [61] and ReCLIP [50].
We conduct experiments following consistent settings with each compared work. To be specific, with the CPT and our codebase, we focus on individual visual prompting performance without post-processing. We use proposals from UNINEXT and MAttNet to demonstrate the robustness of our enhancements. It's important to note that different proposal selections solely affect the box candidates, which are equitably shared among all the comparison prompting methods. Next, with the ReClip codebase, ReClip employs cropping and colorful boxes as visual prompts, with default spatially-relations post-processing. To ensure a fair comparison, we first add cropping as an ensemble to all experiments. Next, to facilitate comparison with RedCircle (which inherently uses Score Subtraction as post-processing and primarily ensembles based on three circle prompts), we adopt the same three types of prompt formats but based on semantic masks. Finally, we explore higher performance possibilities. The preprocessing procedure for text inputs, including the prefix and clear-text principle, remains consistent across all codebases. Additionally, we compare zero-shot
Visual Prompt	PACO	RefCOCO	RefCOCO+	RefCOCOg
Crop	16.5	17.7	21.6	34.3
Keypoint	11.9	16.7	18.5	19.7
Circle	17.4	24.9	29.8	32.4
Colorful Mask	15.2	24.1	21.4	18.6
Blur Reverse Mask	18.3	40.8	44.9	49.6
Figure 6: Ablation study on the NMS threshold and grid size.
Table 4: Accuracy of the part detection with ViT-L on the validation set of each benchmark. The best result is in bold.
Figure 7: Visualization of the candidate grid-wise keypoints, the proposals, the visual prompting image, and the predicted results in part detection.
referring accuracy under unified CLIP models. Unless otherwise stated, all existing method results are reported according to their original papers.
Table 3 shows that the single fine-grained visual prompt, i.e., Blur Reverse Mask can surpass previous works. With a ViT-L model, our FGVP surpasses RedCircle by an average of  $3.4\%$  accuracy in our codebase and  $5.5\% \sim 8.1\%$  with the CPT [61] codebase. Further, we implement FGVP under the ReCLIP [50] codebase, which investigates the ensemble of multiple models, prompts, and post processing. The results show that the improvement brought by FGVP is orthogonal to the ensemble techniques. Under equal comparisons, our FGVP shows better performance than previous works with a consistent gain of an average of  $\sim 3\%$  and a max of  $12.5\%$  in the RefCOCO+ testA set.
4.5 Part Detection
The PACO [44] dataset features annotations for boxes and masks for common objects and their corresponding parts. For the part detection task, models need to locate the part within its object. Different from the referring task, there is no prior information indicating where the parts are. Our proposed pipeline (Fig. 3) is capable of performing zero-shot part detection without box proposals and utilizing only image inputs in these circumstances. Different from the localization keypoints operation employed in RedCircle [40] which only predicts the center location, we instead predict precise semantic masks of the target. Same with the metric in referring expression comprehension, a predicted box is considered correct only when the intersection over union with the ground truth box exceeds 0.5. As shown in Table 4, our proposed method, FGVP, outperforms other coarse prompts for part detection. Notably, we set the grid size to 16 along one side of the image and used an NMS threshold of 0.7 by default. However, better performance can be achieved by including more proposals through the use of a larger grid size and NMS threshold. We conduct an ablation study on them by varying one while fixing the other, as illustrated in Fig. 6. A visualization of the FGVP results is depicted in Fig. 7.
In addition, the referring expression comprehension task can be seen as part detection (each referring instance can be seen as a part of the image) when no prior box proposals are provided. We conducted zero-shot experiments on the validation set of RefCOCO, RefCOCO+, and RefCOCOg datasets, utilizing only images as inputs. As shown in Table 4, the Blur Reverse Mask Prompting without prior box proposals even surpasses certain coarse visual prompting methods with box proposals.
Firstly, FGVP takes longer for inference than other methods because it involves the segmentor to produce semantic masks. To be specific, we compare the inference costs in terms of computation and
Method	SAM scale	Mask-filter	CUDA Memory	Inference Time	IPS	Acc
Crop	-	-	0.91 GB	4.49 min	5.03	45.3
RedCircle [49]	-	-	0.91 GB	4.00 min	5.64	48.9
FGVP (ours)	base	X	1.32 GB	5.20 min	4.34	51.7
✓	1.32 GB	27.47 min	0.82	52.1
large	X	2.14 GB	6.29 min	3.59	51.0
✓	2.14 GB	27.49 min	0.82	52.2
huge	X	3.42 GB	7.34 min	3.08	51.9
✓	3.42 GB	28.02 min	0.81	52.8
Table 5: Comparisons of inference cost. IPS: Image per GPU second.
speed between our method and others. Further, we conduct an ablation study on the scalability of SAM. Notably, the post-processing technique to filter small disconnected regions and holes in masks can further improve performance at the cost of speed. Disabling the mask-filter post-processing will greatly improve the speed without losing too much performance (Table 5). Experiments are run on RefCOCO with a CLIP pre-trained ViT-L/14@336px on  $8 \times$  NVIDIA A100. Above all, speeding up the architecture is an important direction for future improvement.
Secondly, the current implementation does not couple the visual prompt with a specifically designed language prompt. Therefore, the image-text alignment comprehension ability of VLM has not been fully explored appropriately. Another consideration is that there are more possible tasks that can adopt the proposed method for zero-shot transfer, such as referring image segmentation. We leave it to our future work.
In this paper, we focus on the visual prompt, a technique to highlight the target instances in the image content using visible markings such as colorful boxes and circles. The visual prompt is beneficial in invoking potential spatial comprehension within VLMs like CLIP on instance-level tasks. However, existing prompting designs are often too coarse for locating the target instance, leading to unrelated information that may harm performance. Since the topic of visual prompting is rarely explored, we systematically summarize various typical prompting formats and propose Fine-Grained Visual Prompting (FGVP), which utilizes precise semantic masks of target instances derived via Segment Anything (SAM). We discover that the Blur Reverse Mask prompting, which blurs the background, achieves the best performance. Furthermore, we construct two zero-shot classification architectures for regular referring expression comprehension tasks using boxes as prior proposals and for part detection utilizing only input images. We evaluate the effectiveness of FGVP on referring expression comprehension and achieve state-of-the-art (SOTA) performance.
This project is supported by the National Key R&D Program of China (2022ZD0116302), the Young Scientists Fund of the National Natural Science Foundation of China (Grant No. 62206134), the Fundamental Research Funds for the Central Universities (070-63233084), and the Tianjin Key Laboratory of Visual Computing and Intelligent Perception (VCIP). Computation is partially supported by the Supercomputing Center of Nankai University (NKSC). Note that Lingfeng Yang and Jian Yang are with Jiangsu Key Lab of Image and Video Understanding for Social Security, and Key Lab of Intelligent Perception and Systems for High-Dimensional Information of Ministry of Education, Nanjing University of Science and Technology, Nanjing, 210094, P.R. China, while Lingfeng Yang and Xiang Li are with IMPlus@PCALab.
Further research and careful consideration are necessary when utilizing this technology, as the presented proposed method relies on statistics derived from training datasets that may possess biases and could potentially result in negative societal impacts.
[66] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In CVPR, 2021.
[67] Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In CVPR, 2017.
[68] Shiyu Zhao, Zhixing Zhang, Samuel Schulter, Long Zhao, BG Vijay Kumar, Anastasis Stathopoulos, Manmohan Chandraker, and Dimitris N Metaxas. Exploiting unlabeled data with vision and language models for object detection. In ECCV, 2022.
[69] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to prompt for vision-language models. IJCV, 2022.