Recognize Any Regions
Authors: Haosen Yang
Chuofan Ma
Bin Wen
Yi Jiang
Zehuan Yuan
Xiatian Zhu
Venue: NeurIPS 2024 poster (2024)
Keywords: Open Vocabulary Object Recognition;  Zero-shot; Representation Learning
Primary area: machine_vision

Abstract: Understanding the semantics of individual regions or patches of unconstrained images, such as open-world object detection, remains a critical yet challenging task in computer vision. Building on the success of powerful image-level vision-language (ViL) foundation models like CLIP, recent efforts have sought to harness their capabilities by either training a contrastive model from scratch with an extensive collection of region-label pairs or aligning the outputs of a detection model with image-level representations of region proposals. Despite notable progress, these approaches are plagued by computationally intensive training requirements, susceptibility to data noise, 
and deficiency in contextual information. To address these limitations, we explore the synergistic potential of off-the-shelf foundation models, leveraging their respective strengths in localization and semantics. We introduce a novel, generic, and efficient architecture, named RegionSpot, designed to integrate position-aware localization knowledge from a localization foundation model (e.g., SAM) with semantic information from a ViL model (e.g., CLIP). To fully exploit pretrained knowledge while minimizing training overhead, we keep both foundation models frozen, focusing optimization efforts solely on a lightweight attention-based knowledge integration module.
Extensive experiments in open-world object recognition show that our  RegionSpot achieves significant performance gain over prior alternatives, along with substantial computational savings (e.g., training our model with 3 million data in a single day using 8 V100 GPUs). 
RegionSpot outperforms GLIP-L by 2.9 in mAP on LVIS val set,  with an even larger margin of 13.1 AP for more challenging and rare categories, and a 2.5 AP increase on ODinW. Furthermore, it exceeds GroundingDINO-L by 11.0 AP for rare categories on the LVIS minival set.

Recognize Any Regions
Haosen Yang
Chuofan Ma²
Bin Wen
Yi Jiang
Zehuan Yuan
Xiatian Zhu
University of Surrey
The University of Hong Kong
BByteDance
Abstract
Understanding the semantics of individual regions or patches of unconstrained images, such open-world object detection, remains critical yet challenging task in computer vision. Building on the success powerful image- vision- language (ViL) foundation models like CLIP, recent efforts have sought to harness their capabilities by either training contrastive model from scratch with an exten- sive collection of region-label pairs aligning the outputs of detection model with image-level representations of region proposals. Despite notable progress, these approaches are plagued by computationally intensive training requirements, susceptibility to data noise, and deficiency in contextual information. To address these limitations, we explore the synergistic potential of off-the-shelf foundation models, leveraging their respective strengths in localization and semantics. We in- troduce a novel, generic, and efficient architecture, named RegionSpe designed to integrate position-awa aware localization knowledge from localization foundation model (c.g.. SAM) with semantic information from a ViL model (e.g., CLIP). To fully exploit pretrained knowledge while minimizing training overhead, we keep both foundation models frozen, focusing optimization efforts solely on a lightweight attention-base based knowledge integration module. Extensive experiments in open-world object recognition show that our RegionSpot achieves significant performance gain over prior alternatives, along with substantial computational savings (e.g., training our model with 3 million data in a single day using 8 V100 GPUs). RegionSpot outperforms GLIP-L by 2.9 in mAP on LVIS val with even larger margin of 13.1 AP for more challenging and rare categories, and a 2.5 AP increase on ODinW. Furthermore. it exceeds GroundingDINO-L by 11.0 AP for rare categories on the LVIS minival set.
1 Introduction
Remarkable progress has been achieved in the realm of purpose-generic image-level Vision-Language (ViL.) representation learning, exemplified by foundation models like CLIPP 24 and ALIGN (IT). These advancements have led to significant performance improvements across a diverse spectrum of vision and multi-modal downstream tasks 7 44). The efficacy of these approaches can be largely attributed to their utilization of extensive datasets, typically encompassing millions, if not billions, of training samples replete with rich information. In the pursuit of more nuanced approach to visual analysis, researchers have also ventured into the realm of universal region-level (e.g., objects) comprehension. This is evident in recent research endeavors approach to this involves learning the semantics of image regions by applying an image-level pre- trained model (e.g., CLIP) to cropped regions, followed by representational distillation using the output of detection model 7 (3), as depicted in Figure a)). However, utilizing individual cropped
39
43
16
19
22
common
"This work was performed when Haosen Yang worked as an intern at ByteDance. Corresponding authors
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Alignment
Object feature
Feuture map
Reglordput
Concept: Pumplin, Cat, Window
Detedtor
Vi image stoonder
ViL That Encoder
Detector
Teat Encoder
Encoder
Krop
Resice
(a)
(b)
(c)
Figure 1: Illustration of typical region-level visual understanding architecture. (a) Learning the region recognition model by distilling image-level ViL representations from cropped regions and incorporating them into a detection model (e.g., 77]. (b) Fully fine-tuning both vision and
text models with a substantial dataset of region-label pairs. (c) Our proposed approach integrates pretrained (frozen) localization and ViL models, emphasizing the learning of their representational correlation.
regions in this design leads to the loss of crucial contextual information, which can hinder recognition performance. [16) introduced an open-world detector with a fixed ViL model, bypassing knowledge distillation. However, the use of ROIAlign [9] for region feature extraction poses limitations. Furthermore, directly applying an image-level model to isolated local regions is less effective, as the model was pretrained on entire images encompassing both object regions and surrounding context. An alternative, albeit brute-force, approach revolves around constructing region-level representations from scratch, harnessing extensive dataset that pairs regions with labels [18] 38 41 (37 (Figure
Πb)). Nevertheless, this approach grapples with challenges such as the proliferation of noisy pseudo- labels and significant training costs. Furthermore, significant advancements have materialized in the realm of class- class-agnostic visual localization techniques, as illustrated by the notable work of SAM [14. This approach is characterized by its distinctive feature integration of position-aware localization knowledge, which we consider a valuable complement to the inherent capabilities of ViL models. Expanding upon this conceptual framework, our research introduces an innovative architectural paradigm at the region level, herein referred to as RegionSpot. This framework seamlessly incorporates large pre-trained ViL and localization models within efficient training regimen, obviating the necessity for an extensive repository of region-label pairings. Our methodology centers on the acquisition of the correlation between localization data extracted from 'local' regions by the localization model and the semantic representations encompassing the entirety of the image, derived from the ViL model. This strategic approach permits us circumvent the conventional fine-tuning of both pre-trained models-wheree they remain 'frozen' during Kaaning--herell safeguarding the integrity of their rich knowledge and ensuring its maximal utilization, while mitigating the potential for performance degradation. To enact this cross- cross-model correlation, we employ the cross- -ross-attention mechanism [32]. In this configuration, the localization feature assumes the role of the 'query', whereas the ViL feature assumes dual roles as both the 'key" and 'value'. This implementation effectively facilitates the fusion of semantic and localization information manner that is amenable to learning and yields substantive efficacy.
Our contributions are follows: (1) We introduce the concept of integrating off-the-shelf foundation models to tackle region-level visual understanding, (2) To achieve this objective, we introduce a novel architectural paradigm called
which does necessitate training from seratch. This approach excels in both optimization efficiency and data utilization. By circumventing the fine-tuning of both localization and Lasiuu-Language (ViL) components, our architecture retains its openness and adaptability, welcoming the seanless integration of advancements in both domains. Extensive experimentation in the context of open-wont world object understanding confirms the superior performance of our method, even with substantially smaller number of learnable parameters. Remarkably, RegionSpot surpasses the state-of-the- GLIP-L by 2.9 in mAP, with an even more substantial advantage of 13.1 AP observed for the more intricate rare categories.

---

2 Related Work
Zero-shot in image recognition Zero-shot image recognition is the task of recognizing calegories that have not been seen during training. In [5] and [10], the authors utilized visual attributes to facilitate knowledge transfer unfamiliar categories. Researchers have also investigated the utilization of class hierarchies, similarities, and object parts to enhance knowledge transfer, as demonstrated in the works of [ 42 [3]. Recent research has focused on aligning latent image- text embeddings for classifying and describing visual content. [6] pioneered the establishment of a visual semantic space through deep learning. Subsequently, CLIP 24 and ALIGN [T]] attained impressive results via contrastive learning with extensive collections of image-text pairs, showcasing exceptional performance across diverse benchmarks. contrast to previous endeavors that primarily addressed image-leve recognition, we focus fine-grained recognition of visual elements at the regional level.
Zero-shot in region understanding In zero-shot object recognition, the aim is to enable object detectors to identify categories not encountered during training, such as [26]
31 36 Researchers
have explored various methods to bridge the gap between known and unknown categories using
pre-trained semantic or textual features (30] [5 3, knowledge graphs [28] 34] and more. Inspired
by the zero-shot capabilities of Vision-andd-L and-Languagge (ViL) like CLIP (24,, several approaches have
sought to integrate pretrained Vision-and- and-Language (ViL) models. For example, [39 7 4 proposed
method to distill learned image embeddings from CLIP for target detection by focusing on cropped proposal regions. Another approach, RegionCLIP by generating leeudo-labels from captioning data and then proceeds with region- word contrastive pretraining before transferring the knowledge to the detection task. [8] took a novel approach by formulating object detection as grounding problem and incorporating additional grounding data to enhance semantic alignment at both phrase and region levels. Their results demonstrated improved performance, even fully-supervised detection benchmarks. (38) leveraged large-scale image captioning datasets and expanded their knowledge database using generated pseudo-labels, bolstering their detection capabilities. The use of generated pseudo-labels effectively extended the detectors' generalization ability.
4 employs a multistage training strategy. It starts
However, these methods face computational challenges and susceptible to training data incon- sistencies and image-level distractions. Differing from these studies, we explore the synergistic benefits of foundation models SAM [14] and CLIP (24]. Leveraging their strengths in localization and semantics, we propose an innovative region recognition framework.
3 Method
Our objective is to employ efficiently pretrained ViL model and localization model, trained on extensive data, to achieve region-level representation and understanding. These representations facilitate robust object conceptualization, especially for open-world region recognition. To realize this, as shown in Figure (]aa we formulate a new approach, named RegionSpot. In the following sections, we will begin with a brief introduction to the foundational models in Section followed by a comprehensive explanation of our approach with focus on learning region- alignment across two pretrained models in Section 3.2
3.1 Foundation Models
Vision-language foundation models use contrastive learning to map visual and textual data into shared embedding space through a contrastive loss. This technique, exemplified by CLIP with 400 million text-image pairs [24], and ALIGN with 1.8 billion pairs [1I. aims minimize the distances between paired images and texts while maximizing distances between unpaired ones.
Localization foundation models have been advanced significantly. A prominent example is the pioneering SAM model [14, which has been trained on the extensive SA-1B dataset, boasting more than billion automatically generated masks unprecedented scale, surpassing existing segmentation datasets by a factor of 400. This dataset also comprises 11 million images.
SAM comprises three core modules: (a) Image encoder: Utilizing a ViT-based backbone, this module extracts image features, yielding image embeddings. (b) Prompt encoder: It encodes positional information from input points, boxes, masks to facilitate the mask decoder. (c) Mask decoder:
3

---

S Poirtt Product
SAM
Image
Transposed Conv.
Wark Fost
Encoder
Transformer Decoder
Prompt
Feet.
Prompt
Encoder
Mask
Multi-Head
Attention
Region Spot
CLIP
plen
FiF
Aetf-Attettion
plates, bow,, forks, knife, spoon, candle
Text
Yaf
Aliomment Loss
Encoder
Poslf seeare 30lon
(b) Figure 2: Overview of our proposed Regi onSpot. (a) We integrate position-awar tokens from localization model, such as SAM, with image-level feature maps extracted from a ViL model like CLIP. This integration yields region-level semantic tokens, which are then subjected to region text alignment. (b) Our cross- moosl feature interaction design based on the attention mechanism.
(a)
This transformer-bot basee decoder leverages both the extracted image embeddings and prompt tokens to make final mask predictions. One of SAM's remarkable features is its robust zero-shot generalization to novel data, obviating the need for domain-specifer fine-tuning. Thanks to extensive training vast repository of prompt-text pairs, SAM demonstrates exceptional proficiency in object localization.
3.2 Region text alignment with frozen foundation models
In this section. describe how extract position-aware tokens from the localization foundation model and generate image-level semantic features using the ViL foundation model. We achieve inter-model association through cross attss-attention mechanism that facilitates region text alignment.
Region-level position-aware tokens In our approach, we utilize sanually-annotater object bound- ing boxes, denoted as R {ri), ..,, N, as regions of interest in the images. For each of these regions, represented as R, we extract position-aware tokens using the SAM model, denoted as P= [ 1,..,N
As depicted in Figure 2, SAM employs mask decoder to generate a mask based a provided prompt. This process utilizes a transformer decoder, similar to the architecture in DETR [2). to generate an object token. This object token plays crucial role predicting the prompt mask, subsequently predicting dynamic MLP weights and performing point- wise product with the mask features. We refer to this resulting token as "position- awwore" because encodes essential information about the object, including details about its texture and position. Following this, projector is applied map the output dimension the position-awar token the image-level feature space as discussed below.
Image-level semantic feature maps A single image can encompass multiple objects across merous categories, capturing integrated context. We can conceptually view an image's feature map composition of region embeddings with varying structures. To fully capitalize the ViL model, we resize the input image to the required dimensions without cropping. Subsequently, we input this resized image into the ViL model, yielding the image-level semantic feature map denoted V.
Relating position-aware tokens and semantic feature maps Our model, referred to as RegionSpot, efficiently establishes connections between region-level position-awar tokens and image-level semantic feature maps using the cross- cross-attention mechanism [32]. In this mechanism, position-awarr tokens P serve queries, while semantic feature maps V/ take the roles of both keys and values. This relationship is formulated as follows:
F FØK T VC
S Softmax
Vo.
(I)

---

where Fp represents a transformation of P, K2 and Ve are derived from separate linear projections of V. and is the projected feature dimension. This approach, well-established in the literature. has consistently demonstrated its effectiveness in information fusion. In our work, we extend its application to enhance region-level understanding in open-world scenarios. Specifically, we leverage this mechanism to integrate positional information with semantic content extracted from two distinct models at the regional level, while also considering the broader context from the entire image, as depicted in Figure (b).
Loss function In line with prior research, we generate text embeddings by processing category texts along with prompt templates, like a photo of category in the scene, using the text encoder. Then, we perform dot product operation between each semantic token and its corresponding text features to calculate matching scores. These scores can be supervised using the focal loss 20
Zero short inference Following 43), we focus on the more challenging region recognition task by utilizing human-annotated boxes or external region proposal generator. Inheriting the flexible prompting capability from SAM, our model allows for region recognition through prompting.
Experiments
Training data In pursuit of robust training environment, we combined diverse datasets with varying label spaces. Our model's ' flexible architecture allowed us to seamlessly replace one-hot labels with class name strings. For training, we utilized publicly available detection datasets, prising a total of approximately 3 million images. These datasets include Objects 365 (O365) [29].
OpenImages (OI) [15. and V3Det (V3D) 33 each contributing uniquely to the diverse repository.
Objects 365 (0365) is a large-scale object detection dataset featuring 365 distinet object categories across 0.66 million images. Our research employs an enriched version with over 10 million bounding boxes, averaging approximately 15.8 object annotations per image.
Openlmages (OI) currently stands the largest public object detection dataset, encompass- ing about 14.6 million bounding box annotations, equivalent to around annotations per image.
V3Det (V3D) distinguishes itself through hierarchical organization, meticulously structur- ing up to 13,029 categories within a category tree.
Benchmark settings In our rigorous evaluation process, we utilized the extensive LVIS detection dataset 88) which encompasses 1203 categories and 19809 images reserved for validation. We do not prioritize the performance on COCO [21] which includes only 80 common categories covered by the Objects365 training dataset 229) This limitation may not adequately assess model's generalization in an open-world setting.
Since our current emphasis is not object localization, we utilized ground- and class-agnostic bounding boxes from an existing detector to predict categories based on corresponding text descrip- tions, following the RegionCLIP approach [43 Mean Average Precision (mAP) served our evaluation metric.
Implementation details We train RegionSpot using AdamW 13 optimizer with the initial learning rate as 2.5 x 10-5, 5 All models are trained with mini-batch size 16 on 8 GPUs. The default training schedule is 450K iterations, with the learning rate divided by 10 at 350K and 420K iterations. The training process unfolds in two sequential stages: (1) warm-up phase leveraging the Objects365 to initiate the learning of region-wort alignments, and (2) a phase of advanced learning for region-word alignments, utilizing rich compilation from three diverse object detection datasets. The model trained for 450K iterations at each stage. We implement several model variants: (1) snggionSpot-Lttee Integrating the base versions of both SAM and CLIP. (2) Regi RegionSpot-PPro Combining the SAM base with the more extensive CL.IP large architecture. (3) Regi onggitt Prorase: Further extending RegionSpot - -Pro by using input image resolution of 336.
5

---

Table 1: Comparison of open-world zero-shot object recognition performance using ground
(GT) boxes, SAM proposals generate by automatic mask generator, and GLIP boxes on the LVIS
dataset. * indicate finetune the CLIP with Adapter. The training time test on one V100 GPU
Method
Training Data
Proposals
Times
AP,
AP, ;
APall
CLIP-L w/ box
GT
40.6
59.2
48.7
CLIP-L w/ mask
GT
40.8
59.6
49.2
CL3P-L1336 w/ mask
GT
43.2
59.9
49.5
CLIP-L1336* w/ mask
0365, OL V3D
GT
0.30k
46.8
63.2
53.1
RegionSpot-Lite
0365, OI, V3D
GT
0.18k
42.0
65.6
53.0
RegionSpot-Pro
0365, OL, V3D
GT
0.18k
50.6
68.8
56.6
RegionSpot-Pronsa
0365, OI, V3D
GT
0.20k
55.4
68.6
59.9
CLIP-L1336* w/ mask
0365, OI, V3D
SAM
0.30k
11.3
16.4
14.5
RegionSpot-Pro
0365, OI, V3D
SAM
0.18k
13.1
17.3
16.1
RegionSpot-Pronss
0365, OI, V3D
SAM
0.20k
14.3
19.2
18.2
GLIP-T
0365
GLIP-T(B)
57.5k
4.2
13.6
11.3
RegionSpot-Lite
O365
GLIP-T(B)
0.18k
12.7
15.7
14.1
GLIP-T
GolMG,CapMM 0365, OI, V3D
GLIP-T
92.1k
10.1
25.5
17.2
RegionSpot-Lits
GLIP-T
0.18k
20.0
24.2
21.1
GLIP-L
FourODs, Lol.GGllGGGappa
GLIP-L
120k
17.1
35.4
26.9
RegionSpot- Pro133
0365, OI, V3D
GLIP-L
0.2k
30.2
30.0
29.8
Table 2: Evaluation of zero-shot object detection on the LVIS minival dataset.
Method
Training Data
AP,
AP
GLIP-L
CourODs,GoldCo CaMP
28.2
37.3
GroundingDINO--
22.2
33,9
MegionSpot-Pronse
0365,OLV3DET
33.2
36.9
4.1 Zero-shot Inference for Region Recognition
Zero-shot object detection on LVIS Val v1.0 Results on the LVIS benchmark are presented in Table T With ground-truth bounding boxes as region proposals, our model substantially surpasses the CLIP basclines (which applies CLIP on image crops) by large margin (e.g., 48.7, 49.2 v9.59.9). For fair comparison, we finetune the CLIP withe adapter, our model substantially surpasses the CLIP by large margin. Moreover, in simulation of real-world cases, we move forward test method with noisy region proposals generated from off-the-shelf proposal generator. We first employ SAM a proposal generator, inputting dense grid points to automatically generate proposals. It can be seen that RegionSpot still consistently outperforms CLIP (e.g., 14.5 vs. 18.2 on APalt) in this case, demonstrating the robustness of method.
To fully exploit the potential our method and synergy with the advancements of open world object detection (OWD), further utilize region proposals from satte-of-theatt OWD models, i.e., GLIP, as our region prompts. Comparing with GLIP-T trained solely on the objects365 dataset, can observe considerable performance gain achieved by RegionSpot (e.g., 4.2 vs. 12.7 AP, and 11.3 AP vs. 14.1 on APau). After scaling up the training data and use 336 resolution image input, models maintains superior performances over their GLIP counterparts. For instance, www.oospooo--Prsssuuuuuse
GLIP-T by 17.6 AP, with less training data, showing compelling scaling behavior with data scale and input resolution. For more extensive evaluation, we also utilize bounding boxes generated by GLIP-L as prompts. It is noteworthy that RegionSpot achieves an impressive 13.1 increase in AP, compared GLIP-L, even when trained less data at higher efficiency. Despite using a noisy box, we were still able to achieve promising results, thanks to the robust localization ability of SAM. Additional experiments, including the ViLD protocol, can be found in the Appendix
Open vocabulary object detection under ViLD-protocal To thoroughly evaluate our method, we conducted experiments using the ViLD protocol 77. training on base categories and testing on novel ones with the LVIS AP metric. For fair comparsion, all the method training only use the LVIS-base dataset and use the RPN from RegionCL.IP as proposal generator. We also adapted our training to the LVIS-base dataset. As shown in Table 3 RegionSpot demonstrates competitive performance.

---

It outperforms the similarly frozen-backte backbone F-VLM by 1.1 AP, When we compared to Region- CLIP, which benefits from additional caption pretraining. RegionSpot significantly outperforms the pretrained version of RegionCIIIP by 2.6 when utilizing same RPN.
Table 3: Comparison under the ViLD protocol All methods the ResNet50 backbone. *
indicate pre-training with CC-3M
Method
Proposals
Trainable Backbone
AP,
APal all
ViLD
RPN
16.1
22.5
RegionCLIPE
RPN
17.1
28.2
DetDc-ViLD
RPN
17.8
26.8
F-VLM
RPN
X
18.6
24.2
RegionSpot
RPN
X
19.7
25.0
To fully exploit the potential of our method, we report on MiniVal containing 5,000 images introduced in MDETR [12). We use the output proposals from GL.PP as the prompt. As shown Table 2 although we use 9x less training data, method also
Zero-shot object detection on LVIS minival5k 12
model maintains superior performances over GLIP-L by 5.0 APr. Further, surpasses Grounding DINO-L (which even uses a more advanced detector) by 11.0 APr.
Zero-shot instance segmentation We evaluate the performance of instance segmentation in zero-shot setting using the LVIS dataset [8] By leveraging the output from GLIP as the prompt, we direct it to RegionSpot for mask and class prediction. The mask AP is evaluated using the released X-Decoder [45] and OpenSeeD [40], both of which are trained with mask-text pairs. Impressively, as indicated in Table 4 RegionSpp outstrips X-Decoder and OpenSeeD by margins of 14.1 and 3.9 in AP, respectively. These outcomes suggest that our proposed RegionSpot can effectively harness the foundational model's capabilities achieve more accurate region understanding.
Zero-shot object detection on ODinW [17 This benchmark was designed to evaluate model performance in real-world scenarios. To accurately evaluate recognition capabilities, we filter the dataset to include only those with more than three categories. The AP for each dataset is reported in Table 5 Impressively, degionSpot- onSpot-Prosse, utilizing GLIP-L proposals, surpasses GLIP-L by a margin of 2.5 AP, attributable to its precise region recognition. Furthermore, method exceeds the performance of GroundingDINO, even though it employs a more advanced detector.
4.2 Ablation Study
We conducted an ablation study for RegionSpot-BL using the boxes generated by GLIP. Unless otherwise mentioned, training performed three different detection datasets.
Enhancement with CLIP vision embedding We conjugate that key ingredient with degionSpot the use of semantic information from CLIP vision encoder. To validate this assertion, we began evaluation without the CLIP feature and subsequently integrated the class token output from the CLIP vision encoder. Results in Table demonstratt that: (1) The CL.P feature offers significant boost, pushing the baseline without CLIP vision encoder to 22.1, which suggests inherent semantic limitations SAM. (2) More notably, the inclusion CLIP enhances overall performance to 23.7. This underscores the potential of the class token encapsulate global information from the entire image.
Position-aware tokens selection in SAM The position -ware tokens are generated by intermediary module in the SAM. We examined various locations for this generation, specifically after the Prompt encoder, the Transformer decoder, and the MLP within the SAM. Results presented Table 7b indicate that generating output tokens after the Transformer decoder yields the best performance.
Table 4: Evaluation of zero-shot instance segmentation on the LVIS minival dataset.
Method
AP,
AP.
AP,
AP
X-Decoder
9.4
OpenSeed
19.6
RegionSpot-PPe -rrooee
21.5
25.0
23.2
23.5

---

Table 5: Evaluation of zero-shot object detection on the ODinW dataset.
Method
Aerial. Drone.
Aquarium
PascalVOC
shellfish
vehicles
Avg.
GroundingDINO-T
10.3
17.5
55.7
29.5
58.5
34.3
GLIP-T
12.5
18.4
56.2
26.3
56.0
33.8
GLIP-L
7.1
26.9
61.7
68.9
57.3
44.4
Regi RegionSpot-Lig
13.1
20.1
58.2
30.1
57.2
35.7
RegionSpot--Prosse
14.2
27.2
62.7
69.3
61.3
46.9
Table Ablation experiments on LVIS, (a) The effective of CLIP vision encoder; (b) Position- aware tokens selection (c) Depth of RegionSpot.
(a)
(b)
(c)
CLIP
AP
Position-aware tokens
AP
Depth
AP
w/o CLIP vision.
8.0
Prompt encoder
18.6
23.2
+ CLIP feat. map
22.1
Transformer
23.7
3
23.7
+ Class token
23.7
MLP
20.4
6
22.8
This observation is expected since tokens derived from the Prompt encoder are relatively undeveloped. Surprisingly, it outperform GLIP (i.e., 18.6 vs. 17.2). Moreover, there is performance decline after the MLP, which can be attributed to dimensional reduction.
Module architecture Another pivotal aspect of Regi onSoot is the depth of model. To assess its impact, we experimented by varying the depth of our model. As indicated in Table 6c it is imperative for the model to have sufficiently large depth, such as 3 blocks, without being excessive.
Prompt engineering Finally, carried out ablation study focusing prompt engineering, incorporating both box prompts in SAM and text prompts in the text encoder. As evidenced by the results Table 7a (I) Leveraging multiple boxes prompts in SAM boosts performance, achieving an AP of 22.1. 1. This enhancement is credited to the self-attention mechanism of onSpt, which adeptly fuses information from varied regions. (2) Further utilizing text prompts results modest performance boost, specifically an increase of 1.6 AP.
Ablation study of SAM model We conjugate that a key ingredient is the position-awen -wware information from SAM. We evaluating the impact of different SAM model sizes, such ViT-L, is essential. We conducted experiments with varying SAM model sizes. As shown in the Table our findings summarized as follows: (1) Impact of SAM Model Size: Our results indicate that the of larger SAM models (e.g., SAM-L) improves mask AP due to the higher quality of mask generation. However, for box AP, the improvement is not significant. This because the SAM mask token primarily contributes position-awand knowledge, which is already sufficiently captured by ViT-B and ViT-L. (2) Choice of SAM Model: Given focus on region recognition, we opted for SAM-B, balancing performance and computational efficiency.
4.3 Visualization
Result visualization In Figure we present the results of bounding region recognition on the LVIS [8] dataset, comparing between GLIP and RegionSpot. To assess the zero-shot recognition capability, we employ the same bounding boxes for both models. As observed, Regi onSpot can distinguish subtle differences, recognizing smaller objects like "lemon" and "tennis ball" and similar objects like "lantern and "fireplug". Notably, Regi onSoot stands out in terms of the accuracy of its label predictions, especially within the category of rare classes.
Table 7: Ablation experiments on LVIS. (a) The effective of propmpt engineering: (b) The effective
of SAM
(a)
(b)
Prompt
AP.
AP;
AP
SAM
box AP,
mask AP,
bascline
19.6
21.2
18.5
VIT-B
24.9
22.8
w/ mutiple boxes prompt
23.2
25.0
22.1
VIT-I
24.7
23.6
w/ text prompt
24.9
25.5
23.7

---

GLIP
Tarbayo
Figure 3: Qualitative prediction results of GLIP-T [8] (first row) and RegionSpot (second row) on the LVIS dataset 8 Our model recognizes the objects more accurately. Best viewed when zooming-in.
Figure 4: Cross-attention maps in Regi onSot.. These maps show that the position-awar token aligns effectively with the semantic feature map the entire image. In each row, the blue and red boxes are corresponding to the left and right maps respectively.
Model behavior visualization To gain more intuitive understanding the effect brought by Sngio,, we examine the cross attention map on LVIS [. We take the output tokens as 'query' and CLIP feature map 'key" and 'value'. For clearer visualization, the CLIP semantic feature. The resulting attention map clearly depicts the correspondence between the position-aware tokens generated by SAM and the feature map produced by CLIP. Using this arrangement, we gain a visual insight into how RegionSpot establishes connections between distinct features. As depicted in Figure 4 the attention maps vividly showcase Regi ongpot capability to seamlessly incorporate both SAM and CLIP. Such visualizations serve a dual purpose: they highlight the efficacy of our method, and simultaneously, shed light on the intricate mechanisms underpinning the RegionSpot
omit the class token from

---

Conclusions and limitations
In this study, we introduce Regi onSpot, a novel and efficient framework leveraging frozen vision and vision-language foundation models for region recognition, eliminating the need for training from scratch. To fully exploit knowledge in pretrained models and minimize the training overhead, we keep both foundation models frozen and focus optimization efforts solely on a lightweight attention- based knowledge integration module. Extensive experiments in the context of open- open-world object understanding confirms the superior performance of our method, even with a substantially smaller number of learnable parameters, which distinguishes our method and enables efficient training. Impressively, RegionSpot outperforms the leading GLIP--. by 2.9 in mAP, and this lead grows to 13.1 when considering complex rare categories. While our method advances open world region understanding, it still not unleash potential capabilities from the foundmental models, such as the automatic localization ability from SAM, which could reduce reliance external region proposal mechanisms for object detection and enhance versatility, We leave this for further investigation.
References
[1] Zeynep Akata, Mateusz Malinowski, Mario Fritz, and Bernt Schiele. Multi-cue zero-shot learning with strong supervision. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 59-68, 2016.
[2] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229. Springer, 2020.
[3] Soravit Changpinyo, Wei-Lun Chao, and Fei Sha. Predicting visual exemplars of unseen classes for zero-shot learning. In Proceedings of the IEEE international conference on computer vision, pages 3476-3485, 2017.
[4] Yu Du, Fangyun Wei, Zihe Zhang, Miaojing Shi, Yue Gao, and Guoqi Li. Learning to prompt for open-vocabulary object detection with vision-language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14084-14093, 2022.
[5] Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In 2009 IEEE conference on computer vision and pattern recognition, pages 1778-178 1785. IEEE, 2009.
[6] Andrea Frome, Greg S Corrado, Jon Shlens, Samy Bengio, Jeff Dean, Mare' Aurelio Ranzato, and Tomas Mikolov. Devise: A deep risual-semantic embedding model. Advances in neural information processing systems, 26, 2013.
[7] Xiuye Gu, Tsung-Yo Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.13921 2021.
[8] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5356-5364, 2019.
[9] Kaiming Hc, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 2961-2969, 2017.
[10] Dinesh Jayaraman and Kristen Grauman. Zero-shot recognition with unreliable attributes. Advances in neural information processing systems, 27, 2014.
[11] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun- Hin-Hsuan Sung. Zhen Li, and Tom Duerig. Scaling up visual and lision-language representation learning with noisy text supervision. In International conference on machine learning, pages 4904 4916. PMLR, 2021.
[12] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1780-1790, 2021.
10

---

[13] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[14] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yet Lo, al. Segment anything. arXiv preprint arXiv:2304.0066. 2023.
[15] Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-EI-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings, Stefan Popov, Andreas Veit, al. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://github. com/openimages, 283):18, 2017.
[16] Weicheng Kuo, Yin Cui, Xiuye Gu, AJ Piergiovanni, and Anelia Angelova. F-vlm: Open- vocabulary object detection upon frozen vision and language models. arXiv preprint 0xXiv:2209.15633 2022.
[17] Chunyuan Li, Haotian Liu, Liunian Li, Pengchuan Zhang, Jyoti Aneja, Jianwei Yang, Ping Jin, Houdong Hu, Zicheng Liu, Yong Jae Lee, et al. Elevater: A benchmark and toolkit for evaluating language-augmented visual models. Advances in Neural Information Processing Systems, 35:9287-9301, 2022.
[18] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10965-10975, 2022.
[19] Chuang Lin, Peize Sun, Yi Jiang, Ping Luo, Lizhen Qu, Gholamreza Haffari, Zehuan Yuan, and Jianfei Cai. Learning object-language alignments for open-vocabulary object detection. arXiv 14843, preprint arXiv:2211. 148443, 2022.
[20] Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages 2980-2988, 2017.
[21] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[22] Chuofan Ma, Jiang, Xin Wen, Zehuan Yuan, and Xiaojuan Qi. Codet: Co-occurrence guided region-word alignment for open- vecabbocay object detection. arXiv preprint arXiv:2310.16667, 2023.
[23] Zongyang Ma, Guan Luo, Jin Gao, Liang Li, Yuxin Chen, Shaoru Wang, Congxuan Zhang, and Weiming Hu. Open-vocabulary one-stage detection with hierarchical visual-langure knowledge distillation. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages
2022.
[24] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8888 8763. PMLR, 2021.
[25] Scott Reed, Zeynep Akata, Honglak Lee, and Bernt Schiele. Learning deep representations of fine-grained visual descriptions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 49 08,,5,850 2016.
[26] Shaoqing Ren, Kaiming He. Ross Girshick, and Jian Sun. Faster r-cnn: Towards tile object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[27] Marcus Rohrbach, Michael Stark, and Bernt Schiele. Evaluating knowledge transfer and zero-shot learning in large-scale setting. In CVPR 2011, pages 1641-1648. IEEE, 201 L.
11

---

[28] Ruslan Salakhutdinov, Antonio Torralba, and Josh Tenenbaum. Learning to share visual appearance for multiclass object detection. In CVPR 2011, pages 1481-1488. IEEE, 2011.
[29] Shuai Shao, Zeming Li, Tianyuan Zhang. Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. 0bjects365: A large-scale. high-quality dataset for object detection. In Proceedings of the IEEE/CVF international conference on computer vision, pages
2019.
[30] Richard Socher, Milind Ganjoo, Christopher D Manning, and Andrew Ng. Zero-shot learning through cross-modal transfer. Advances in neural information processing systems, 26, 2013.
[31] Peize Sun, Rufeng Zhang, Yi Jiang, Tao Kong. Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-end object detection with learnable proposals. In Proceedings of the IEEE/CVV conference on computer vision and pattern recognition, pages 14454-14463, 2021.
[32] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[33] Jiaqi Wang, Pan Zhang, Tao Chu, Yuhang Cao, Yujie Zhou, Tong Wu, Bin Wang, Con- ghui He, and Dahua Lin. V3det: Vast vocabulary visual detection dataset. arXiv preprini arXiv:2304.03752, 2023.
[34] Xiaolong Wang, Yufei Ye, and Abhinav Gupta. Zero-shot recognition via semantic embeddings and knowledge graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6857-6866, 2018
[35] Guo-Sen Xie, Li Liu, Fan Zhu, Fang Zhao, Zheng Zhang, Yazhou Yao, Jie Qin, and Ling Shao. Region graph embedding network for zero-shot learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part IV 16, pages 562- 360. Springer, 2020.
[36] Bin Yan, Yi Jiang, Jiannan Wu, Dong Wang, Ping Luo, Zehuan Yuan, and Huchuan Lu. Universal instance perception object discovery and retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15325-15336, 2023.
[37] Lewei Yao, Jianhua Han, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, and Hang Xu. Detclipy2: Scalable open-vocabulary object detection pre-training via word-region alignment. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 23497-235066 2023.
[38] Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detclip: vectionary-enriche visual-concept paralleled pre-training for open-world detection. Advances in Neural Information Processing Systems, 35:9125-995 29339188. 2022.
[39] Yuhang Zang, Wei Li, Kaiyang Zhou, Chen Huang, and Chen Change Loy. Open-vocabulary detr with conditional matching. In European Conference on Computer Vision, pages 102-122. Springer, 2022.
[40] Hao Zhang, Feng Li, Xueyan Zou, Shilong Liu, Chunyuan Li, Jianfeng Gao, Jianwei Yang, and Lei Zhang. A simple framework for open-vocabulary segmentation and detection. arXiv preprint arXiv:2303.08833 2023.
[41] Haotian Zhang. Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq Hwang, and Jianfeng Gao. Glipy2: Unifying localization and lanione understanding. Advances in Neurul Information Processing Systems, 35:36067-36080. 2022.
[42] Hang Zhao, Xavier Puig, Bolei Zhou, Sanja Fidler, and Antonio Torralba. Open vocabulary scene parsing. In Proceedings of the IEEE International Conference on Computer Vision, pages 2002-2010, 2017.
12

---

[43] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionelip: Region- based language-image pretraining. In Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, pages 16793-16803, 2022.
[44] Kaiyang Zhou, Jingkang Yang. Chen Change Loy, and Ziwei Liu. Conditional prompt learning for rision-languae models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16816-16825, 2022.
[45] Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behi, Jianfeng Wang, Lu Yuan, al. Generalized decoding for pixel, image, and language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15116-15127, 2023.
13

---

A Additional Experimental Result
Training efficiency. To illustrate the training efficiency of Regionnppl we benchmark its GPU training hours against RegionCLIP and GLIP, as showcased in Table Even though we utilize the ViT-Large as our backbone, our model achieves faster training. This efficiency can be attributed to our frozen approach and processing of images at reduced resolution of 224x224 for the CLIP Large. All benchmarks were executed in a consistent hardware environment, leveraging eight NVIDIA V100 GPUs. In stark contrast, GLIP necessitates an extensive 92K GPU hours, whopping 436 times more than our approach, mainly due its exhaustive fine-tuning of both Vision and Text models. Interestingly, even when RegionCLIP adopts smaller backbone akin to ours, it still requires 4.6K GPU hours.
Table 8: Comparisons with the training efficiency.
Train time (GPU hours)
Learnable Param (M)
Method
Training data
RegionCLIP
CC3M O365, GoldG, Cap4M FourODs,Golde GoldGGCa22M O365,01, D.5,OL..GOOG..CAMM..OOO Cap4M.COCO.RRIIC 0365. O1, V3D
4.6K
GLIP-T GLIP-L
92.1K
120K
289
GDINO-L
no released
341
RegionSpot-Pro
0.2K
35
Table 9: Effect of increasing the detection training data.
Data
AP,
AP.
AP,
AP
Objects365
11.9
13.6
20.2
15.9
Objects365 + Openlmages
16.4
18.2
23.7
20.1
Objects365 + OpenImages V3DET
24.9
21.6
25.5
23.7
Benefit increasing the detection training data Table showcases the performance improve- ments observed when augmenting the training data size. Through our proposed framework, integrating additional detection data from diverse sources consistently enhances the capabilities rooted pre- trained knowledge. Compared to training with only Objects365, including Openlmages effectively improves the overall AP from 15.9 to 20.1. The inclusion of V3Det further propels the performance, achieving impressive overall AP of 23.7. This improvement is particularly significant for rare categories, with an increase from 16.4 to 24.9 (a gain +8.5 AP), attributable to its extensive vocabulary.
B
More visualizations on LVIS
Figure 5 provides more examples on LVIS 8
14

---

GLIP
OURS
524
GLIP
csr TnLoOTONE C
oURS
Figure 5: More visualizations comparison with GLIP. Best viewed when zoomed-in
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Refer to the Abstract and Introduction in Sec. and Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Refer to Sec. 5
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
No theoretical results in this paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: The result is reproducible by training the framework under our instructions.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

Answer: [No]
Justification:
The code will be available after being accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https guwdes/ddeeeeee Cooeeeoodeesssseeeeee 7/nips.com / ssiniioiscyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Refer to the Implement Details in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: expensive.
The error bars are not reported because it would be too computationally
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
18

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Refer to the Implement Details and Inference Strategy in Sec.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Refer to the Broader Impacts in Sec.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The code and models and will be released do not have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The assets properly credited and the license and terms of use explicitly mentioned and properly respected in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
20

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "rtMpqce0Ne", "reviewer": "Reviewer_TiX5", "summary": "The paper introduces RegionSpot, a compute-efficient method that leverages localization foundation models (such as SAM) with semantic information from a ViL model (such as CLIP). RegionSpot is demonstrated on multiple scenarios and achieve better results than baseline methods while being much faster to train than others.", "questions": "1. Table 1: While I understand the drop in performance when going from GT boxes to SAM proposals and GLIP boxes -- I am wondering if there is a way to evaluate the results on the basis of classification criterion then detection? RegionSpot is primarily about naming region. Penalizing it for a stricter detection may not be a correct idea.", "limitations": "Not explicitly stated.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Computational Efficiency: RegionSpot does not require large computational resources to train region identification model. RegionSpot keeps both foundation models (SAM and CLIP) frozen, focusing optimization efforts solely on a light weight attention-based knowledge integration module.\n\n2. Encoding region-level and image-level visual knowledge with text-annotations: RegionSpot cleverly utilizes region-level knowledge from SAM and it use image-level information with ViL (such as CLIP). This allows RegionSpot to capture more contextual information as compared to other baseline methods.\n\n3. The paper presents diverse analysis showing the effectiveness of proposed method, RegionSpot.", "weaknesses": "1. RegionSpot use SAM. Effectively, RegionSpot could be used for object detection by optimizing SAM properly. In its current version, RegionSpot is restricted to identifying regions given a region proposal or a bounding box.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful comments.\n\n**Q1: RegionSpot use SAM. Effectively, RegionSpot could be used for object detection by optimizing SAM properly. RegionSpot is restricted to identifying regions given a regional proposal or a bounding box.**\n\n**R1:** Many thanks for these great comments. We highlight the following points:\n\n1. Our method preserves SAM's flexible prompting capability and pretrained knowledge by keeping SAM frozen instead of fine-tuning it to save costs. This approach allows RegionSpot to maintain interactive region recognition, extract region-specific semantic features, perform open vocabulary object detection, and effectively segment regions.\n2. Similar to our approach, recent works like RegionCLIP[1] and DetPro[2] also use external region proposals, as recognizing regions of interest is the focus for all. As demonstrated in ViLD[3], F-VLM[4], and Groma[5], existing regional proposal methods are already highly adaptable and directly applicable across different domains. However, recognizing the detected regions presents more challenges and is the bottleneck. That is why we focus on addressing this aspect. We will further stress.\n\n[1] RegionCLIP: Region-based Language-Image Pretraining CVPR2022\n\n[2] DetPro: Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Model CVPR2022\n\n[3] ViLD: Open-vocabulary Object Detection via Vision and Language Knowledge Distillation ICLR2022\n\n[4] F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models ICLR2023\n\n[5] Groma: Grounded Multimodal Large Language Model with Localized Visual Tokenization ECCV2024\n\n**Q2: More metrics to evaluate RegionSpot.**\n\n**R2:** Many thanks. As suggested, we use the classification metric, Accuracy, to evaluate RegionSpot. Same as the main paper, we use masks to crop the region. As shown in Table 1, our models maintain superior performances over the CLIP baseline by a large margin. We will add this evaluation.\n\nTable 1: Comparison with Accuracy metric, * indicate finetune the CLIP with Adapter.\n\n|                        | proposals | Accuray |\n| ---------------------- | --------- | ------- |\n| CLIP-L_↑336 w/ mask_  | GT        | 44.7%   |\n| CLIP-L_↑336* w/ mask_ | GT        | 58.1%   |\n| RegionSpot-Pro _↑336_ | GT        | 68.2%  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rqpcKE3F6x", "reviewer": "Reviewer_4JVL", "summary": "The paper proposed a method for open-world object detection, which utilises the segment anything model (SAM) to produce region priors and the CLIP model to extract image and language features. The region priors from SAM, which are implicitly encoded in the query tokens, are used in a learnable transformer decoder to perform cross-attention with the image-level features extracted using CLIP. The decoder is trained in a contrastive manner such that the query tokens are matched with the corresponding language embeddings. The proposed method demonstrated strong performance on the challenging LVIS dataset, with less training time.", "questions": "1. What is the training time in Table 1 measured by?\n\n2. What is the inference speed of the model?\n\n3. How would the model compare against SAM itself? As I understand, the segmentation masks could be easily converted into bounding boxes based on the boundary pixels. Furthermore, segmentation is essentially a harder task than object detection. As SAM already has the capability of detecting objects (characterised by masks instead of boxes) using different types of prompts, what is the advantage of the proposed method?", "limitations": "The authors did not discuss the inference speed of the proposed model, which I suspect will be a significant issue as it employs two foundation models.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method combines the regions priors from the SAM model with the feature extraction capability of the CLIP model, and eliminates the need of training a region proposal network, which is shown to speed up the training process.\n\n2. A opposed to cropping out image regions that contain objects and extracting region features, the proposed method exploits the implicit region priors from SAM and use the corresponding query tokens to produce detections. This method only computes the image-level features once and eliminates the repetitive computation caused by overlapping bounding boxes, analogous to the improvement from R-CNN to Fast R-CNN.", "weaknesses": "1. The main advantage of the proposed method seems to be the low training time, which is somewhat less important compared to inference speed. The proposed model employs two foundation models, which will most likely result in very slow inference speed. Yet the paper did not include any details around this.\n\n2. The paper could benefit from some more insights on what kind of region priors the position-aware tokens encode. For instance, object detection models such as conditional-DETR and DAB-DETR have revealed that the interaction between the queries and the positional embeddings of the image features is key to localising the object. As such, one would expect that the position-aware tokens may have high similarity with the sinusoidal positional embeddings around the regions that contain the object.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your insightful comments.\n\n**Q1: The main advantage of the proposed method seems to be the low training time, which is somewhat less important compared to inference speed. The proposed model employs two foundation models, which will most likely result in very slow inference speed. Yet the paper did not include any details around this.**\n\nR1: Apologies for the omission of inference speed data. To demonstrate the efficiency of our proposed RegionSpot, we compared the training and inference speeds of RegionSpot with GroundingDINO-T[1] and GLIP-T[2] on the zero-shot object detection benchmark on LVIS val v1.0. We analyzed model performance, training time in GPU hours, and inference latency using the same hardware, NVIDIA V100 GPU. Since GroundingDINO does not provide the training code or LVIS evaluation code, we only tested latency by modifying their code, referring to the provided simple inference code.\n\nAs shown in Table 1, compared to GLIP, we achieve a 460x speed-up in training time. Additionally, RegionSpot achieves 6.5 FPS (0.15 s/image) on a single V100 during inference on LVIS, including all component processes like RPN, SAM, and CLIP. In contrast, GLIP-T and GroundingDINO-T achieve only 0.2 FPS (5 s/image) and 0.14 FPS (7.1 s/image), respectively, due to their visual-text concept alignment through sequential formulation and early fusion. Despite using two foundation models, our inference speed outperforms GLIP and GroundingDINO due to: (1) low-resolution inputs for CLIP, (2) parallel region-text token formulation from CLIP, (3) parallel multi-prompt processing in the SAM decoder, (4) a lightweight decoder, and (5) a faster RPN proposal generator. Note that in the paper, for fair comparison, we used the same proposals as GLIP. But one can utilize any proposal generator in practice. The above revision would improve our work clearly thanks to the reviewer’s feedback.\n\nTable 1: Efficiency comparison on LVIS val v1.0.\n\n|Method|Training(GPU Hours)|Inference(FPS)|AP_r|\n|-|-|-|-|\n|Grounding DINO-T[1]|-|0.2|-|\n|GLIP-T[2]|92.1K|0.14|10.1|\n|RegionSpot+RPN|0.2K|6.5|14.2|\n\n[1] Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection ECCV2024\n\n[2] GLIP: Grounded Language-Image Pre-training CVPR 2022\n\n**Q2: The paper could benefit from some more insights on what kind of region priors the position-aware tokens encode. For instance, object detection models such as conditional-DETR and DAB-DETR have revealed that the interaction between the queries and the positional embeddings of the image features is key to localising the object. As such, one would expect that the position-aware tokens may have high similarity with the sinusoidal positional embeddings around the regions that contain the object.**\n\nR2: Thank you for these insightful comments. We agree with the reviewer on the importance of understanding the kind of region prior information encoded by the position-aware tokens. To explore this, we conducted an ablation study analyzing output tokens from SAM at various locations. The study concluded that generating the output token after the Transformer decoder yields the best performance, as it not only encodes the coordinate position information but also incorporates semantic position information. As illustrated in Figure 4 of the main paper, we also visualized the similarity between the position-aware token and the CLIP feature. As expected, the regions containing objects in the CLIP feature showed higher similarity. We appreciate the reviewer's suggestion and have included the above discussion in the final version.\n\n**Q3: What is the training time in Table 1 measured by?**\n\nR3: We measured the training time in GPU hours using the same V100 hardware. We will further clarify this in the final version.\n\n**Q4: How would the model compare against SAM itself? As I understand, the segmentation masks could be easily converted into bounding boxes based on the boundary pixels. Furthermore, segmentation is essentially a harder task than object detection. As SAM already has the capability of detecting objects (characterised by masks instead of boxes) using different types of prompts, what is the advantage of the proposed method?**\n\nR4: Thanks for your question. We agree that segmentation is a harder task than object detection. However, SAM only supports simple visual prompts, such as the points and boxes, and outputs the class-agnostic mask.\n\nIf we want to achieve the class-aware mask, one way is to pass the segmented region to CLIP to achieve the zero-shot region recognition. However, utilizing individual cropped regions only leads to the loss of crucial contextual information, which can hinder recognition performance. Meanwhile,  there is often a big gap between the current task (i.e., region level understanding) and pretrain task (i.e., image level understanding). In our method, instead of cropping the regions from an image, we utilize mask tokens from SAM, which has strong position-aware information，to find the corresponding semantic details from the ViL feature map, enhancing the semantic understanding at a regional level. Our method can unleash the pretrained power from the pretrained foundation model without the need for training from scratch as required by the previous works [1]. Our design not only achieves superior region recognition accuracy but also is more efficient computationally and in training data collection.\n\n[1] RegionCLIP: Region-based Language-Image Pretraining CVPR2022", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J4mbJCxy4w", "reviewer": "Reviewer_AWre", "summary": "To address open-world object detection, this paper proposes RegionSpot, which combines the localization capabilities of SAM with the classification strengths of CLIP. RegionSpot integrates position-aware tokens from SAM with image-level feature maps extracted from CLIP, creating region-level semantic tokens. These tokens are then aligned with text representations to enhance recognition accuracy. RegionSpot achieves state-of-the-art performance on the LVIS and ODinW benchmarks.", "questions": "1. Figure 4 illustrates that the \"position-aware token aligns effectively with the semantic feature map of the entire image.\" How do the tokens from SAM contain more effective localization features compared to tokens from other localization models, such as Mask-RCNN or CAM from CLIP? In other words, why was SAM chosen over other localization models?\n2. Instead of using the pipeline of RegionSpot, if I obtain all masks using SAM, forming a bounding box for each instance, then use CLIP to conduct zero-shot classification for detection, how are the results? And why the design of RegionSpot could outperform such a baseline?", "limitations": "Yes. The authors claim that \"While our method advances open world region understanding, it still (does) not unleash potential capabilities from the fundamental models, such as the automatic localization ability from SAM, which could reduce reliance on external region proposal mechanisms for object detection and enhance versatility.\"", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The approach of forming position-aware tokens from SAM, and the way they interact with CLIP features are innovative. These tokens, containing localization features, should enhance detection capabilities.\n2. RegionSpot demonstrates substantial performance improvements compared to other methods and baselines across various settings.", "weaknesses": "1. While effective, the method by which RegionSpot uses position-aware tokens is somewhat implicit. It is not entirely clear how these localization features directly contribute to the performance gains.\n2. Combining CLIP and SAM for detection, although effective, is relatively straightforward. Despite some non-trivial modifications, the overall novelty of the approach may be perceived as limited.\n3. A small typo in line 166 \"Zero short inference\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your insightful comments.\n\n**Q1: While effective, the method by which RegionSpot uses position-aware tokens is somewhat implicit. It is not entirely clear how these localization features directly contribute to performance gains.**\n\nR1: We summarize the reason for performance gain again. RegionSpot utilized the position-aware token to query the semantic information extracted from a ViL model since the position token already learned the position-aware information about the region from the SAM pretrained model and context information outside of individual regions, which could provide extra cues for helping recognition while aligning with CLIP features. Meanwhile, the ViL feature map contains semantic information about the whole image. Hence, they only need light weight connectors that can bridge the region level position-aware information and image level semantic information to achieve the open world region level understanding. Additionally, we did the ablation to verify that different location tokens from SAM present different effects. Please check Table 6 (b) in the main paper. We will further claim these in the final version.\n\n**Q2: Combining CLIP and SAM for detection, although effective, is relatively straightforward. Despite some non-trivial modifications, the overall novelty of the approach may be perceived as limited.**\n\nR2: Apologies for this misunderstanding. It has been a trending research focus in AI towards integrating rich, pre-trained models to enhance performance. The motivations are manifold, e.g., using pre-trained models saves computational power and time, making powerful models accessible without extensive resources. This is responsible, green, and scalable. Our work falls in this realm. We note that our novelty is not introducing new architectures (e.g., cross attention) or simply assembling a pipeline of SAM and CLIP which is our baseline in Table 1 of the main paper. Instead, our key idea is to optimize efficient open-world region understanding by leveraging existing foundational models. In this study, we use the ViL foundational model CLIP and the localization foundational model SAM to validate our approach. We verify the effectiveness of RegionSpot through extensive testing on various tasks and datasets in a zero-shot manner. Additionally, we conducted ablation studies to explore how to fully leverage the pretrained capabilities of these foundational models. We will further clarify these points in the final version.\n\n**Q3: Figure 4 illustrates that the \"position-aware token aligns effectively with the semantic feature map of the entire image.\" How do the tokens from SAM contain more effective localization features compared to tokens from other localization models, such as Mask-RCNN or CAM from CLIP? In other words, why was SAM chosen over other localization models?**\n\nR3: Great question, thanks. Although Mask-RCNN or CAM from CLIP only has the coarse ability to identify the object, they are generally inferior in localizing the objects in the wild. Contrastly, SAM trained with billion scale prompt-mask pairs, is capable of segmenting/localizing a wide range of visual structures in diverse scenarios, by taking a prompt consisting of points, a bounding box as input. Its zero-shot segmentation abilities have led to a rapid paradigm shift. As SAM adopts the DETR architecture, the prompt token from the SAM already has the position aware information to query the object. Hence we choose it as the localization foundation model. But our method is not limited to SAM, also can apply other SAM-like models, such as HQ-SAM[1] and SAM 2 [2]. We will further clarify.\n\n[1] HQ-SAM: Segment Anything in High Quality NIPS2023\n\n[2] SAM 2: Segment Anything in Images and Videos arxiv\n\n**Q4: Instead of using the pipeline of RegionSpot, if I obtain all masks using SAM, forming a bounding box for each instance, then use CLIP to conduct zero-shot classification for detection, how are the results? And why the design of RegionSpot could outperform such a baseline?**\n\nR4: In our submission, we have already provided this suggested baseline obtaining SAM output mask and then crop the region part feed to CLIP in Table 1. Instead of individual cropping regions from an image, RegionSpot uniquely uses the position aware token from SAM to query the corresponding semantic feature from the whole-image CLIP feature map using the cross attention. This enables RegionSpot to model both object content within each region and context information outside the region – the latter cannot be leveraged by this SAM+CLIP baseline. That explains why our method excels .\n\n**Q5: Typo mistakes.**\n\nR5: Thanks, we will fix all in the revision.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lOTNSIU8mx", "reviewer": "Reviewer_B3PH", "summary": "The paper presents RegionSpot, a novel architecture designed for efficient open-world visual region recognition. The primary goal of RegionSpot is to leverage the strengths of powerful pretrained foundation models, specifically a localization model (SAM) and a vision-language model (CLIP), to improve the recognition of individual regions or patches within images. RegionSpot focuses on keeping both foundation models frozen and optimizing only a lightweight attention-based knowledge integration module. This results in significant computational savings and reduced training time. Extensive experiments demonstrate that RegionSpot outperforms state-of-the-art models, showing substantial gains in mean Average Precision (mAP) and especially excelling in recognizing rare and challenging categories.", "questions": "1. This paper lacks a detailed comparison with the state-of-the-art models, and does not provide reference links for the methods used in the comparison,\t\n\n2. Can you provide more detailed information on the computational resources required for training and inference? How do these requirements compare to other state-of-the-art models?\n\n3. Have you analyzed the failure cases where RegionSpot did not perform well? What were the common reasons for these failures?\n\n4. How significant is the impact of keeping the foundation models frozen during training? Have you experimented with fine-tuning these models to assess any potential performance gains?\n\n5. Have you explored the impact of using different types of prompts or additional features in your model?", "limitations": "1. The authors should provide a more detailed discussion on the limitations of relying on specific pretrained models. They could explore scenarios where these models might fail or underperform, such as in completely novel domains or with significantly different data distributions.\n\n2. Identifying and discussing potential performance bottlenecks within the RegionSpot architecture would be beneficial. This includes the integration module and cross-attention mechanism, which might limit scalability or introduce latency in real-time applications.\n\n3. The evaluation is conducted on a few specific benchmarks. Broader evaluation across different datasets and tasks would provide a more comprehensive understanding of the model's capabilities and limitations.\n\n4. The method relies on external region proposals or ground truth bounding boxes. Integrating end-to-end learning for region proposal and classification could further improve efficiency.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The combination of SAM and CLIP in a frozen state with a lightweight attention-based module is a unique approach that leverages pre-existing models' strengths.\n\nThe approach significantly reduces training time and computational resources, making it more practical for real-world applications.\n\nThe model demonstrates substantial improvements over previous methods, particularly in recognizing rare and challenging categories.", "weaknesses": "1. Dependency on Pretrained Models: The approach is heavily reliant on the quality and capabilities of the SAM and CLIP models. Any inherent limitations or biases in these models could impact RegionSpot's performance.\n\n2. Lack of Novel Algorithmic Innovations:  While RegionSpot's integration of SAM and CLIP is innovative, the methodology itself does not introduce fundamentally new algorithms or theoretical advancements in the field of computer vision. The primary contribution lies in the effective use of existing models rather than developing new techniques or algorithms.\n\n3. Absence of New Training Paradigms: The approach focuses on combining pretrained models in a novel way but does not offer new training paradigms or optimization strategies. This could be seen as a limitation in terms of pushing the boundaries of current methodologies.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for your insightful comments.\n\n**Q1: Dependency on Pretrained Models.**\n\nR1: Many thanks for these great comments. It has been a trending research focus in AI towards integrating rich, pre-trained models to enhance a target task, particularly when these foundation models get stronger and also heavier [1-3]. The motivations are manifold, e.g., using pre-trained models saves computational power and time, making powerful models accessible without extensive resources. This is responsible, green, and scalable. Our work falls in this realm. Additionally, RegionSpot is not limited to CLIP and SAM in general, though they are selected in our implementation due to their salient performances. RegionSpot is very flexible for integrating more advanced ViL and localization foundation models, such as InternVL[4], SAM 2[5].\n\n[1]BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models ICML2023\n\n[2]Visual Instruction Tuning NIPS2023\n\n[3]Adding Conditional Control to Text-to-Image Diffusion Models ICCV2023\n\n[4]InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks CVPR2024\n\n[5]SAM 2: Segment Anything in Images and Videos arxiv\n\n**Q2: Lack of Novel Algorithmic Innovations and Absence of New Training Paradigms**\n\nR2:  Introducing fundamentally new architectures/networks and training paradigms is one format of algorithmic innovation but not the only way. We consider our approach of leveraging existing foundation models for achieving superior region recognition to be significant as fine-grained region understanding is an essential requirement in computer vision. This also advocates reuse of computing resources, avoiding the need to develop task-specific foundation models. So we argue this work matters.\n\n**Q3: This paper lacks a detailed comparison with the state-of-the-art models, and does not provide reference links for the methods used in the comparison.**\n\nR3: We have provided detailed comparisons of training data, training time, learnable parameters and performance in the Table 1, 2 and appendix. We will further include additional information, such as inference time, in the final version. Additionally, we will add reference links to every table.\n\n**Q4: Can you provide more detailed information on the computational resources required for training and inference? How do these requirements compare to other state-of-the-art models?**\n\nR4: Thanks. We have provided the training data and training time in Table 1 and appendix.\n\nTo demonstrate the efficiency of our proposed RegionSpot, we compared the training and inference speeds of RegionSpot with GroundingDINO-T and GLIP-T  on the zero-shot object detection benchmark on LVIS. We analyzed model performance, training time in GPU hours, and inference latency using the same hardware, NVIDIA V100 GPU. Since GroundingDINO does not provide the training code and LVIS evaluation code, we only tested latency by modifying their code, referring to the provided simple inference code.\n\nAs shown in Table 1, compared to GLIP, we achieve a 460x speed-up in training time. Additionally, RegionSpot achieves 6.5 FPS (0.15 s/image) on a single V100 during inference on LVIS, including all component processes like RPN, SAM, and CLIP. In contrast, GLIP-T and GroundingDINO-T achieve only 0.2 FPS (5 s/image) and 0.14 FPS (7.1 s/image), respectively, due to their visual-text concept alignment through sequential formulation and early fusion. Despite using two foundation models, our inference speed outperforms GLIP and GroundingDINO due to: (1) low-resolution inputs for CLIP, (2) parallel region-text token formulation from CLIP, (3) parallel multi-prompt processing in the SAM decoder, (4) a lightweight decoder, and (5) a faster RPN proposal generator. Note that in the paper, for fair comparison, we used the same proposals as GLIP. However, one can utilize any proposal generator in practice. The above revision would improve our work clearly thanks to the reviewer’s feedback. We will clarify.\n\nTable 1: Efficiency comparison on LVIS val v1.0.\n|Method|Training(GPU Hours)|Inference(FPS)|AP_r|\n|-|-|-|-|\n|Grounding DINO-T[1]|-|0.2|-|\n|GLIP-T[2]|92.1K|0.14|10.1|\n|RegionSpot+RPN|0.2K|6.5|14.2|\n\n[1]Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection ECCV2024\n\n[2]Grounded Language-Image Pre-training CVPR 2022\n\n**Q5: Have you analyzed the failure cases where RegionSpot did not perform well? What were the common reasons for these failures?**\n\nR5: Great question. As shown in the Table 1, the recognition ability drops when going from ground-truth boxes to SAM proposals or GLIP boxes, same for all the region recognition methods including ours. That means the accuracy of object localization still matters to performance, though existing localization methods are already strong with some room out there for further improvement. We will add more visualization analysis.\n\n**Q6: How significant is the impact of keeping the foundation models frozen during training? Have you experimented with fine-tuning these models to assess any potential performance gains?**\n\nR6: Great question. Fine-tuning these foundation models is not attempted as that will break the zero-shot segmentation ability from SAM due to catastrophic forgetting, as verified by HQ-SAM[1] and F-VLM[2] , along with getting the training process more resource intensive. We will clarify.\n\n[1]Segment Anything in High Quality NIPS2023\n\n[2]F-VLM: Open-Vocabulary Object Detection upon Frozen Vision and Language Models ICLR2023\n\n**Q7: Have you explored the impact of using different types of prompts or additional features in your model?**\n\nR7: Great question. We have performed prompt engineering ablation study as shown in Table 7(a) in the main paper. Also, we explored different CLIP feature styles and position-aware tokens from SAM to verify RegionSpot in Table 6 (a) and (b) in the main paper. We will further highlight.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel and efficient method for open-world visual region recognition, leveraging the strengths of pre-trained foundation models like SAM and CLIP.  
The reviews are positive given the paper's computational efficiency, innovative approach, and strong empirical results, particularly in recognizing rare and challenging object categories.  
While some concerns were raised regarding the reliance on pre-trained models and the lack of novel algorithmic contributions, the authors' rebuttal effectively addressed these points, highlighting the practical significance of their approach and its potential for broader impact.  
Considering the positive reviews, the authors' thorough responses, I recommend accepting this paper.
Decision: Accept (poster)