Recognize Any Regions

Authors: haosen  yang
chuofan  ma
bin  wen
yi  jiang
zehuan  yuan
xiatian  zhu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 03:17:46

Abstract: Understanding the semantics of individual regions or patches of unconstrained images, such as open-world object detection, remains a critical yet challenging task in computer vision. Building on the success of powerful image-level vision-language (ViL) foundation models like CLIP, recent efforts have sought to harness their capabilities by either training a contrastive model from scratch with an extensive collection of region-label pairs or aligning the outputs of a detection model with image-level representations of region proposals. Despite notable progress, these approaches are plagued by computationally intensive training requirements, susceptibility to data noise, 
and deficiency in contextual information. To address these limitations, we explore the synergistic potential of off-the-shelf foundation models, leveraging their respective strengths in localization and semantics. We introduce a novel, generic, and efficient architecture, named RegionSpot, designed to integrate position-aware localization knowledge from a localization foundation model (e.g., SAM) with semantic information from a ViL model (e.g., CLIP). To fully exploit pretrained knowledge while minimizing training overhead, we keep both foundation models frozen, focusing optimization efforts solely on a lightweight attention-based knowledge integration module.
Extensive experiments in open-world object recognition show that our  RegionSpot achieves significant performance gain over prior alternatives, along with substantial computational savings (e.g., training our model with 3 million data in a single day using 8 V100 GPUs). 
RegionSpot outperforms GLIP-L by 2.9 in mAP on LVIS val set,  with an even larger margin of 13.1 AP for more challenging and rare categories, and a 2.5 AP increase on ODinW. Furthermore, it exceeds GroundingDINO-L by 11.0 AP for rare categories on the LVIS minival set.

Main paper: 
 introduction: 
 Remarkable progress has been achieved in the realm of purpose-generic image-level Vision-Language (ViL) representation learning, as exemplified by foundation models like CLIP [24] and ALIGN [11] . These advancements have led to significant performance improvements across a diverse spectrum of vision and multi-modal downstream tasks [7, 44] . The efficacy of these approaches can be largely attributed to their utilization of extensive datasets, typically encompassing millions, if not billions, of training samples replete with rich information. In the pursuit of a more nuanced approach to visual analysis, researchers have also ventured into the realm of universal region-level (e.g., objects) comprehension. This is evident in recent research endeavors [7, 39, 23, 4, 43, 16, 19, 22] . A common approach to this involves learning the semantics of image regions by applying an image-level pretrained model (e.g., CLIP) to cropped regions, followed by representational distillation using the output of a detection model [7, 39] , as depicted in Figure 1 and incorporating them into a detection model (e.g., [7] ). (b) Fully fine-tuning both vision and text models with a substantial dataset of region-label pairs. (c) Our proposed approach integrates pretrained (frozen) localization and ViL models, emphasizing the learning of their representational correlation.regions in this design leads to the loss of crucial contextual information, which can hinder recognition performance. [16] introduced an open-world detector with a fixed ViL model, bypassing knowledge distillation. However, the use of ROIAlign [9] for region feature extraction poses limitations. Furthermore, directly applying an image-level model to isolated local regions is less effective, as the model was pretrained on entire images encompassing both object regions and surrounding context. An alternative, albeit brute-force, approach revolves around constructing region-level representations from scratch, harnessing an extensive dataset that pairs regions with labels [18, 38, 41, 37] (Figure 1 (b)). Nevertheless, this approach grapples with challenges such as the proliferation of noisy pseudolabels and significant training costs. Furthermore, significant advancements have materialized in the realm of class-agnostic visual localization techniques, as illustrated by the notable work of SAM [14] . This approach is characterized by its distinctive feature-an integration of position-aware localization knowledge, which we consider a valuable complement to the inherent capabilities of ViL models. Expanding upon this conceptual framework, our research introduces an innovative architectural paradigm at the region level, herein referred to as RegionSpot. This framework seamlessly incorporates large pre-trained ViL and localization models within an efficient training regimen, obviating the necessity for an extensive repository of region-label pairings. Our methodology centers on the acquisition of the correlation between localization data extracted from 'local' regions by the localization model and the semantic representations encompassing the entirety of the image, derived from the ViL model. This strategic approach permits us to circumvent the conventional fine-tuning of both pre-trained models-wherein they remain 'frozen' during training-thereby safeguarding the integrity of their rich knowledge and ensuring its maximal utilization, all while mitigating the potential for performance degradation. To enact this cross-model correlation, we employ the cross-attention mechanism [32] . In this configuration, the localization feature assumes the role of the 'query', whereas the ViL feature assumes dual roles as both the 'key' and 'value'. This implementation effectively facilitates the fusion of semantic and localization information in a manner that is amenable to learning and yields substantive efficacy.Our contributions are as follows: (1) We introduce the concept of integrating off-the-shelf foundation models to tackle region-level visual understanding. (2) To achieve this objective, we introduce a novel architectural paradigm called RegionSpot, which does not necessitate training from scratch. This approach excels in both optimization efficiency and data utilization. By circumventing the fine-tuning of both localization and Vision-Language (ViL) components, our architecture retains its openness and adaptability, welcoming the seamless integration of advancements in both domains. Extensive experimentation in the context of open-world object understanding confirms the superior performance of our method, even with a substantially smaller number of learnable parameters. Remarkably, RegionSpot surpasses the state-of-the-art GLIP-L by 2.9 in mAP, with an even more substantial advantage of 13.1 AP observed for the more intricate rare categories. 
related work: 
 Zero-shot in image recognition Zero-shot image recognition is the task of recognizing categories that have not been seen during training. In [5] and [10] , the authors utilized visual attributes to facilitate knowledge transfer to unfamiliar categories. Researchers have also investigated the utilization of class hierarchies, similarities, and object parts to enhance knowledge transfer, as demonstrated in the works of [27, 1, 42, 35] . Recent research has focused on aligning latent imagetext embeddings for classifying and describing visual content. [6] pioneered the establishment of a visual semantic space through deep learning. Subsequently, CLIP [24] and ALIGN [11] attained impressive results via contrastive learning with extensive collections of image-text pairs, showcasing exceptional performance across diverse benchmarks. In contrast to previous endeavors that primarily addressed image-level recognition, we focus on fine-grained recognition of visual elements at the regional level.In zero-shot object recognition, the aim is to enable object detectors to identify categories not encountered during training, such as [26, 31, 36] . Researchers have explored various methods to bridge the gap between known and unknown categories using pre-trained semantic or textual features [30, 25, 3] , knowledge graphs [28, 34] , and more. Inspired by the zero-shot capabilities of Vision-and-Language (ViL) like CLIP [24] , several approaches have sought to integrate pretrained Vision-and-Language (ViL) models. For example, [39, 7, 4] proposed a method to distill learned image embeddings from CLIP for target detection by focusing on cropped proposal regions. Another approach, RegionCLIP [43] employs a multistage training strategy. It starts by generating pseudo-labels from captioning data and then proceeds with region-word contrastive pretraining before transferring the knowledge to the detection task. [18] took a novel approach by formulating object detection as a grounding problem and incorporating additional grounding data to enhance semantic alignment at both phrase and region levels. Their results demonstrated improved performance, even on fully-supervised detection benchmarks. [38] leveraged large-scale image captioning datasets and expanded their knowledge database using generated pseudo-labels, bolstering their detection capabilities. The use of generated pseudo-labels effectively extended the detectors' generalization ability.However, these methods face computational challenges and are susceptible to training data inconsistencies and image-level distractions. Differing from these studies, we explore the synergistic benefits of foundation models SAM [14] and CLIP [24] . Leveraging their strengths in localization and semantics, we propose an innovative region recognition framework.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Refer to Sec. 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 Our objective is to employ efficiently a pretrained ViL model and a localization model, trained on extensive data, to achieve region-level representation and understanding. These representations facilitate robust object conceptualization, especially for open-world region recognition. To realize this, as shown in Figure 2 (a) we formulate a new approach, named RegionSpot. In the following sections, we will begin with a brief introduction to the foundational models in Section 3.1, followed by a comprehensive explanation of our approach with focus on learning region-text alignment across two pretrained models in Section 3.2.Vision-language foundation models use contrastive learning to map visual and textual data into a shared embedding space through a contrastive loss. This technique, exemplified by CLIP with 400 million text-image pairs [24] , and ALIGN with 1.8 billion pairs [11] , aims to minimize the distances between paired images and texts while maximizing distances between unpaired ones. Localization foundation models have been advanced significantly. A prominent example is the pioneering SAM model [14] , which has been trained on the extensive SA-1B dataset, boasting more than 1 billion automatically generated masks-an unprecedented scale, surpassing existing segmentation datasets by a factor of 400. This dataset also comprises 11 million images. This transformer-based decoder leverages both the extracted image embeddings and prompt tokens to make final mask predictions. One of SAM's remarkable features is its robust zero-shot generalization to novel data, obviating the need for domain-specific fine-tuning. Thanks to extensive training on a vast repository of prompt-text pairs, SAM demonstrates exceptional proficiency in object localization.In this section, we describe how we extract position-aware tokens from the localization foundation model and generate image-level semantic features using the ViL foundation model. We achieve inter-model association through a cross-attention mechanism that facilitates region text alignment.Region-level position-aware tokens In our approach, we utilize manually-annotated object bounding boxes, denoted as R = {r i }, i = 1, .., N , as regions of interest in the images. For each of these regions, represented as R, we extract position-aware tokens using the SAM model, denoted asP = {p i }, i = 1, .., N .As depicted in Figure 2 , SAM employs a mask decoder to generate a mask based on a provided prompt. This process utilizes a transformer decoder, similar to the architecture in DETR [2] , to generate an object token. This object token plays a crucial role in predicting the prompt mask, subsequently predicting dynamic MLP weights and performing a point-wise product with the mask features. We refer to this resulting token as "position-aware" because it encodes essential information about the object, including details about its texture and position. Following this, a projector is applied to map the output dimension of the position-aware token to the image-level feature space as discussed below.Image-level semantic feature maps A single image can encompass multiple objects across numerous categories, capturing integrated context. We can conceptually view an image's feature map as a composition of region embeddings with varying structures. To fully capitalize the ViL model, we resize the input image to the required dimensions without cropping. Subsequently, we input this resized image into the ViL model, yielding the image-level semantic feature map denoted as V .Relating position-aware tokens and semantic feature maps Our model, referred to as RegionSpot, efficiently establishes connections between region-level position-aware tokens and image-level semantic feature maps using the cross-attention mechanism [32] . In this mechanism, position-aware tokens P serve as queries, while semantic feature maps V take on the roles of both keys and values. This relationship is formulated as follows:EQUATIONwhere F p represents a transformation of P , K v and V v are derived from separate linear projections of V , and C is the projected feature dimension. This approach, well-established in the literature, has consistently demonstrated its effectiveness in information fusion. In our work, we extend its application to enhance region-level understanding in open-world scenarios. Specifically, we leverage this mechanism to integrate positional information with semantic content extracted from two distinct models at the regional level, while also considering the broader context from the entire image, as depicted in Figure 2(b) .In line with prior research, we generate text embeddings by processing category texts along with prompt templates, like a photo of category in the scene, using the text encoder. Then, we perform a dot product operation between each semantic token and its corresponding text features to calculate matching scores. These scores can be supervised using the focal loss [20] .Zero short inference Following [43] , we focus on the more challenging region recognition task by utilizing human-annotated boxes or external region proposal generator. Inheriting the flexible prompting capability from SAM, our model allows for region recognition through prompting.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [NA]Answer: [No] Justification: The code will be available after being accepted.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experiments: 
 Training data In pursuit of a robust training environment, we combined diverse datasets with varying label spaces. Our model's flexible architecture allowed us to seamlessly replace one-hot labels with class name strings. For training, we utilized publicly available detection datasets, comprising a total of approximately 3 million images. These datasets include Objects 365 (O365) [29] , OpenImages (OI) [15] , and V3Det (V3D) [33] , each contributing uniquely to the diverse repository.• Objects 365 (O365) is a large-scale object detection dataset featuring 365 distinct object categories across 0.66 million images. Our research employs an enriched version with over 10 million bounding boxes, averaging approximately 15.8 object annotations per image.• OpenImages (OI) currently stands as the largest public object detection dataset, encompassing about 14.6 million bounding box annotations, equivalent to around 8 annotations per image.• V3Det (V3D) distinguishes itself through a hierarchical organization, meticulously structuring up to 13,029 categories within a category tree.Benchmark settings In our rigorous evaluation process, we utilized the extensive LVIS detection dataset [8] , which encompasses 1203 categories and 19809 images reserved for validation. We do not prioritize the performance on COCO [21] which includes only 80 common categories covered by the Objects365 training dataset [29] . This limitation may not adequately assess a model's generalization in an open-world setting.Since our current emphasis is not on object localization, we utilized ground-truth and class-agnostic bounding boxes from an existing detector to predict categories based on corresponding text descriptions, following the RegionCLIP approach [43] . Mean Average Precision (mAP) served as our evaluation metric.Implementation details We train RegionSpot using AdamW [13] optimizer with the initial learning rate as 2. For fair comparison, we finetune the CLIP withe adapter, our model substantially surpasses the CLIP by a large margin. Moreover, in simulation of real-world cases, we move forward to test our method with noisy region proposals generated from off-the-shelf proposal generator. We first employ SAM as a proposal generator, inputting dense grid points to automatically generate proposals. It can be seen that RegionSpot still consistently outperforms CLIP (e.g., 14.5 vs.18.2 on AP all ) in this case, demonstrating the robustness of our method.To fully exploit the potential of our method and synergy with the advancements of open world object detection (OWD), we further utilize region proposals from state-of-the-art OWD models, i.e., GLIP, as our region prompts. Comparing with GLIP-T trained solely on the objects365 dataset, we can observe a considerable performance gain achieved by RegionSpot (e.g., 4.2 vs.12.7 on AP r and 11.3 AP vs.14.1 on AP all ). After scaling up the training data and use 336 resolution image as input, our models maintains superior performances over their GLIP counterparts. For instance, RegionSpot-Pro↑336surpasses GLIP-T by 17.6 AP r with less training data, showing compelling scaling behavior with data scale and input resolution. For more extensive evaluation, we also utilize bounding boxes generated by GLIP-L as prompts. It is noteworthy that RegionSpot achieves an impressive 13.1 increase in AP r compared to GLIP-L, even when trained on less data at higher efficiency. Despite using a noisy box, we were still able to achieve promising results, thanks to the robust localization ability of SAM. Additional experiments, including the ViLD protocol, can be found in the AppendixTo thoroughly evaluate our method, we conducted experiments using the ViLD protocol [7] , training on base categories and testing on novel ones with the LVIS AP metric. For fair comparsion, all the method training only use the LVIS-base dataset and use the RPN from RegionCLIP as proposal generator. We also adapted our training to the LVIS-base dataset. As shown in Table 3 , RegionSpot demonstrates competitive performance.It outperforms the similarly frozen-backbone F-VLM by 1.1 AP r . When we compared to Region-CLIP, which benefits from additional caption pretraining, RegionSpot significantly outperforms the pretrained version of RegionCLIP by 2.6 when utilizing same RPN. Zero-shot object detection on LVIS minival5k [12] To fully exploit the potential of our method, we report on MiniVal containing 5,000 images introduced in MDETR [12] . We use the output proposals from GLIP as the prompt. As shown in Table 2 , although we use 9x less training data, our model maintains superior performances over GLIP-L by 5.0 on APr. Further, our method also surpasses Grounding DINO-L (which even uses a more advanced detector) by 11.0 in APr.We evaluate the performance of instance segmentation in a zero-shot setting using the LVIS dataset [8] . By leveraging the output from GLIP as the prompt, we direct it to RegionSpot for mask and class prediction. The mask AP is evaluated using the released X-Decoder [45] and OpenSeeD [40] , both of which are trained with mask-text pairs. Impressively, as indicated in Table 4 , RegionSpot outstrips X-Decoder and OpenSeeD by margins of 14.1 and 3.9 in AP, respectively. These outcomes suggest that our proposed RegionSpot can effectively harness the foundational model's capabilities to achieve more accurate region understanding.Zero-shot object detection on ODinW [17] This benchmark was designed to evaluate model performance in real-world scenarios. To accurately evaluate recognition capabilities, we filter the dataset to include only those with more than three categories. The AP for each dataset is reported in Table 5 . Impressively, RegionSpot-Pro↑336, utilizing GLIP-L proposals, surpasses GLIP-L by a margin of 2.5 AP, attributable to its precise region recognition. Furthermore, our method exceeds the performance of GroundingDINO, even though it employs a more advanced detector.We conducted an ablation study for RegionSpot-BL using the boxes generated by GLIP. Unless otherwise mentioned, training was performed on three different detection datasets.We conjugate that a key ingredient with RegionSpot is the use of semantic information from CLIP vision encoder. To validate this assertion, we began our evaluation without the CLIP feature and subsequently integrated the class token output from the CLIP vision encoder. Results in This observation is expected since tokens derived from the Prompt encoder are relatively undeveloped. Surprisingly, it can outperform GLIP (i.e., 18.6 vs. 17.2). Moreover, there is a performance decline after the MLP, which can be attributed to dimensional reduction.Module architecture Another pivotal aspect of RegionSpot is the depth of model. To assess its impact, we experimented by varying the depth of our model. As indicated in Table 6c , it is imperative for the model to have a sufficiently large depth, such as 3 blocks, without being excessive.Prompt engineering Finally, we carried out an ablation study focusing on prompt engineering, incorporating both box prompts in SAM and text prompts in the text encoder. As evidenced by the results in Table 7a : (1) Leveraging multiple boxes as prompts in SAM boosts performance, achieving an AP of 22.1. This enhancement is credited to the self-attention mechanism of RegionSpot, which adeptly fuses information from varied regions. ( 2) Further utilizing text prompts results in a modest performance boost, specifically an increase of 1.6 AP.Ablation study of SAM model We conjugate that a key ingredient is the position-aware information from SAM. We evaluating the impact of different SAM model sizes, such as ViT-L, is essential.We conducted experiments with varying SAM model sizes. As shown in the Table 7b , our findings are summarized as follows: (1) Impact of SAM Model Size: Our results indicate that the use of larger SAM models (e.g., SAM-L) improves mask AP due to the higher quality of mask generation. However, for box AP, the improvement is not significant. This is because the SAM mask token primarily contributes position-aware knowledge, which is already sufficiently captured by ViT-B and ViT-L. (2) Choice of SAM Model: Given our focus on region recognition, we opted for SAM-B, balancing performance and computational efficiency.Result visualization In Figure 3 , we present the results of bounding region recognition on the LVIS [8] dataset, comparing between GLIP and RegionSpot. To assess the zero-shot recognition capability, we employ the same bounding boxes for both models. As observed, RegionSpot can distinguish even subtle differences, recognizing smaller objects like "lemon" and "tennis ball" and similar objects like "lantern" and "fireplug". Notably, RegionSpot stands out in terms of the accuracy of its label predictions, especially within the category of rare classes. To gain more intuitive understanding on the effect brought by our RegionSpot, we examine the cross attention map on LVIS [8] . We take the output tokens as 'query' and CLIP feature map as 'key' and 'value'. For clearer visualization, we omit the class token from the CLIP semantic feature. The resulting attention map clearly depicts the correspondence between the position-aware tokens generated by SAM and the feature map produced by CLIP. Using this arrangement, we gain a visual insight into how RegionSpot establishes connections between distinct features. As depicted in Figure 4 , the attention maps vividly showcase RegionSpot capability to seamlessly incorporate both SAM and CLIP. Such visualizations serve a dual purpose: they highlight the efficacy of our method, and simultaneously, shed light on the intricate mechanisms underpinning the RegionSpot. 
conclusions and limitations: 
 In this study, we introduce RegionSpot, a novel and efficient framework leveraging frozen vision and vision-language foundation models for region recognition, eliminating the need for training from scratch. To fully exploit knowledge in pretrained models and minimize the training overhead, we keep both foundation models frozen and focus optimization efforts solely on a lightweight attentionbased knowledge integration module. Extensive experiments in the context of open-world object understanding confirms the superior performance of our method, even with a substantially smaller number of learnable parameters, which distinguishes our method and enables efficient training. Impressively, RegionSpot outperforms the leading GLIP-L by 2.9 in mAP, and this lead grows to 13.1 when considering complex rare categories. While our method advances open world region understanding, it still not unleash potential capabilities from the foundmental models, such as the automatic localization ability from SAM, which could reduce reliance on external region proposal mechanisms for object detection and enhance versatility. We leave this for further investigation. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: Refer to the Implement Details in Sec. 4.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: The error bars are not reported because it would be too computationally expensive.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: Refer to the Broader Impacts in Sec. 5.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification: The code and models we use and will be released do not have high risk for misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: The assets are properly credited and the license and terms of use explicitly mentioned and properly respected in this paper.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: Training efficiency. To illustrate the training efficiency of RegionSpot, we benchmark its GPU training hours against RegionCLIP and GLIP, as showcased in Table 8 . Even though we utilize the ViT-Large as our backbone, our model achieves faster training. This efficiency can be attributed to our frozen approach and processing of images at a reduced resolution of 224x224 for the CLIP Large. All benchmarks were executed in a consistent hardware environment, leveraging eight NVIDIA V100 GPUs. In stark contrast, GLIP necessitates an extensive 92K GPU hours, a whopping 436 times more than our approach, mainly due to its exhaustive fine-tuning of both Vision and Text models. Interestingly, even when RegionCLIP adopts a smaller backbone akin to ours, it still requires 4.6K GPU hours. Benefit of increasing the detection training data Table 9 showcases the performance improvements observed when augmenting the training data size. Through our proposed framework, integrating additional detection data from diverse sources consistently enhances the capabilities rooted in pretrained knowledge. Compared to training with only Objects365, including OpenImages effectively improves the overall AP from 15.9 to 20.1. The inclusion of V3Det further propels the performance, achieving an impressive overall AP of 23.7. This improvement is particularly significant for rare categories, with an increase from 16.4 to 24.9 (a gain of +8.5 AP), attributable to its extensive vocabulary.Figure 5 provides more examples on LVIS [8] . • The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The paper proposes a novel and efficient method for open-world visual region recognition, leveraging the strengths of pre-trained foundation models like SAM and CLIP.  
The reviews are positive given the paper's computational efficiency, innovative approach, and strong empirical results, particularly in recognizing rare and challenging object categories.  
While some concerns were raised regarding the reliance on pre-trained models and the lack of novel algorithmic contributions, the authors' rebuttal effectively addressed these points, highlighting the practical significance of their approach and its potential for broader impact.  
Considering the positive reviews, the authors' thorough responses, I recommend accepting this paper.

Reviews: review: paper_summary: paper_summary: The paper introduces RegionSpot, a compute-efficient method that leverages localization foundation models (such as SAM) with semantic information from a ViL model (such as CLIP). RegionSpot is demonstrated on multiple scenarios and achieve better results than baseline methods while being much faster to train than others.
strength_weakness: strength_weakness: 1. RegionSpot use SAM. Effectively, RegionSpot could be used for object detection by optimizing SAM properly. In its current version, RegionSpot is restricted to identifying regions given a region proposal or a bounding box.
questions: questions: 1. Table 1: While I understand the drop in performance when going from GT boxes to SAM proposals and GLIP boxes -- I am wondering if there is a way to evaluate the results on the basis of classification criterion then detection? RegionSpot is primarily about naming region. Penalizing it for a stricter detection may not be a correct idea.
limitations: limitations: Not explicitly stated.
paper_summary: paper_summary: The paper proposed a method for open-world object detection, which utilises the segment anything model (SAM) to produce region priors and the CLIP model to extract image and language features. The region priors from SAM, which are implicitly encoded in the query tokens, are used in a learnable transformer decoder to perform cross-attention with the image-level features extracted using CLIP. The decoder is trained in a contrastive manner such that the query tokens are matched with the corresponding language embeddings. The proposed method demonstrated strong performance on the challenging LVIS dataset, with less training time.
strength_weakness: strength_weakness: 1. The main advantage of the proposed method seems to be the low training time, which is somewhat less important compared to inference speed. The proposed model employs two foundation models, which will most likely result in very slow inference speed. Yet the paper did not include any details around this.

2. The paper could benefit from some more insights on what kind of region priors the position-aware tokens encode. For instance, object detection models such as conditional-DETR and DAB-DETR have revealed that the interaction between the queries and the positional embeddings of the image features is key to localising the object. As such, one would expect that the position-aware tokens may have high similarity with the sinusoidal positional embeddings around the regions that contain the object.
questions: questions: 1. What is the training time in Table 1 measured by?

2. What is the inference speed of the model?

3. How would the model compare against SAM itself? As I understand, the segmentation masks could be easily converted into bounding boxes based on the boundary pixels. Furthermore, segmentation is essentially a harder task than object detection. As SAM already has the capability of detecting objects (characterised by masks instead of boxes) using different types of prompts, what is the advantage of the proposed method?
limitations: limitations: The authors did not discuss the inference speed of the proposed model, which I suspect will be a significant issue as it employs two foundation models.
paper_summary: paper_summary: To address open-world object detection, this paper proposes RegionSpot, which combines the localization capabilities of SAM with the classification strengths of CLIP. RegionSpot integrates position-aware tokens from SAM with image-level feature maps extracted from CLIP, creating region-level semantic tokens. These tokens are then aligned with text representations to enhance recognition accuracy. RegionSpot achieves state-of-the-art performance on the LVIS and ODinW benchmarks.
strength_weakness: strength_weakness: 1. While effective, the method by which RegionSpot uses position-aware tokens is somewhat implicit. It is not entirely clear how these localization features directly contribute to the performance gains.
2. Combining CLIP and SAM for detection, although effective, is relatively straightforward. Despite some non-trivial modifications, the overall novelty of the approach may be perceived as limited.
3. A small typo in line 166 "Zero short inference"
questions: questions: 1. Figure 4 illustrates that the "position-aware token aligns effectively with the semantic feature map of the entire image." How do the tokens from SAM contain more effective localization features compared to tokens from other localization models, such as Mask-RCNN or CAM from CLIP? In other words, why was SAM chosen over other localization models?
2. Instead of using the pipeline of RegionSpot, if I obtain all masks using SAM, forming a bounding box for each instance, then use CLIP to conduct zero-shot classification for detection, how are the results? And why the design of RegionSpot could outperform such a baseline?
limitations: limitations: Yes. The authors claim that "While our method advances open world region understanding, it still (does) not unleash potential capabilities from the fundamental models, such as the automatic localization ability from SAM, which could reduce reliance on external region proposal mechanisms for object detection and enhance versatility."
paper_summary: paper_summary: The paper presents RegionSpot, a novel architecture designed for efficient open-world visual region recognition. The primary goal of RegionSpot is to leverage the strengths of powerful pretrained foundation models, specifically a localization model (SAM) and a vision-language model (CLIP), to improve the recognition of individual regions or patches within images. RegionSpot focuses on keeping both foundation models frozen and optimizing only a lightweight attention-based knowledge integration module. This results in significant computational savings and reduced training time. Extensive experiments demonstrate that RegionSpot outperforms state-of-the-art models, showing substantial gains in mean Average Precision (mAP) and especially excelling in recognizing rare and challenging categories.
strength_weakness: strength_weakness: 1. Dependency on Pretrained Models: The approach is heavily reliant on the quality and capabilities of the SAM and CLIP models. Any inherent limitations or biases in these models could impact RegionSpot's performance.

2. Lack of Novel Algorithmic Innovations:  While RegionSpot's integration of SAM and CLIP is innovative, the methodology itself does not introduce fundamentally new algorithms or theoretical advancements in the field of computer vision. The primary contribution lies in the effective use of existing models rather than developing new techniques or algorithms.

3. Absence of New Training Paradigms: The approach focuses on combining pretrained models in a novel way but does not offer new training paradigms or optimization strategies. This could be seen as a limitation in terms of pushing the boundaries of current methodologies.
questions: questions: 1. This paper lacks a detailed comparison with the state-of-the-art models, and does not provide reference links for the methods used in the comparison,	

2. Can you provide more detailed information on the computational resources required for training and inference? How do these requirements compare to other state-of-the-art models?

3. Have you analyzed the failure cases where RegionSpot did not perform well? What were the common reasons for these failures?

4. How significant is the impact of keeping the foundation models frozen during training? Have you experimented with fine-tuning these models to assess any potential performance gains?

5. Have you explored the impact of using different types of prompts or additional features in your model?
limitations: limitations: 1. The authors should provide a more detailed discussion on the limitations of relying on specific pretrained models. They could explore scenarios where these models might fail or underperform, such as in completely novel domains or with significantly different data distributions.

2. Identifying and discussing potential performance bottlenecks within the RegionSpot architecture would be beneficial. This includes the integration module and cross-attention mechanism, which might limit scalability or introduce latency in real-time applications.

3. The evaluation is conducted on a few specific benchmarks. Broader evaluation across different datasets and tasks would provide a more comprehensive understanding of the model's capabilities and limitations.

4. The method relies on external region proposals or ground truth bounding boxes. Integrating end-to-end learning for region proposal and classification could further improve efficiency.

Comments: title: thanks!
Response to authors
Response to authors
comment: We thank the reviewer for recommending acceptance. We appreciate the valuable feedback and will incorporate it into the final version of our paper.
Thanks for the rebuttal. It is very helpful.
Dear Reviewer,

Thank you for your time and expertise in reviewing our paper and participating in the rebuttal process. Your feedback has greatly improved our work. We hope our rebuttal has addressed your concerns. If any issues remain, we are ready to discuss them further. Given the review timeline, we would appreciate your prompt review of our revised responses. If our clarifications have resolved your concerns, we kindly request reconsideration of the initial rating. If not, we welcome further discussion. Thank you again for your thoughtful consideration.
Dear Reviewer,

Thank you for your time and expertise in reviewing our paper and participating in the rebuttal process. Your feedback has greatly improved our work. We hope our rebuttal has addressed your concerns. If any issues remain, we are ready to discuss them further. Given the review timeline, we would appreciate your prompt review of our revised responses. If our clarifications have resolved your concerns, we kindly request reconsideration of the initial rating. If not, we welcome further discussion. Thank you again for your thoughtful consideration.
We thank the reviewer for recommending acceptance. We appreciate the constructive discussion and will incorporate it into our final version.
I have thoroughly read the authors' responses and the comments from other reviewers. Thank you for the detailed answers to my questions. I am willing to upgrade my vote to "Borderline accept."
Thank you for confirming that your concerns have been largely addressed. We appreciate your willingness to engage in further discussion and are here to provide any additional information or clarification you may need.
I've thoroughly reviewed the authors' responses and appreciate their thoughtful engagement. Most of my concerns have been addressed. I will stay in touch for further discussion as we approach the final rating.