LOVD: Large Open Vocabulary  Object detection

Authors: cora  pbovd
vldet  regionclip
lovd  𝑉
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-25 06:31:46

Abstract: Existing open-vocabulary object detectors require an accurate and compact vocabulary pre-defined during inference. Their performance is largely degraded in real scenarios where the underlying vocabulary may be indeterminate and often exponentially large. To have a more comprehensive understanding of this phenomenon, we propose a new setting called Large-and-Open Vocabulary object Detection, which simulates real scenarios by testing detectors with large vocabularies containing thousands of unseen categories. The vast unseen categories inevitably lead to an increase in category distractors, severely impeding the recognition process and leading to unsatisfactory detection results. To address this challenge, We propose a Large and Open Vocabulary Detector (LOVD) with two core components, termed the Image-to-Region Filtering (IRF) module and Cross-View Verification (CV$^2$) scheme. To relieve the category distractors of the given large vocabularies, IRF performs image-level recognition to build a compact vocabulary relevant to the image scene out of the large input vocabulary, followed by region-level classification upon the compact vocabulary. CV$^2$ further enhances the IRF by conducting image-to-region filtering in both global and local views and produces the final detection categories through a multi-branch voting mechanism. Compared to the prior works, our LOVD is more scalable and robust to large input vocabularies, and can be seamlessly integrated with predominant detection methods to improve their open-vocabulary performance. Source code will be made publicly available.

Main paper: 
 introduction: 
 Open-vocabulary object detection [49] advances conventional detectors with the capability to recognize and localize novel object categories that are unseen during training. Prevalent open-vocabulary detectors are often backend by Vision-Language (VL) models [33] pre-trained on large-scale unlabeled image-text pairs. They mostly consist of the category-agnostic detection followed by the openvocabulary classification, where the latter amounts to a cross-modal matching between visual and textual features in the aligned feature space. Though promising progress has been achieved, existing methods entail a precise and compact input vocabulary as a prerequisite for inference, where the object categories in the vocabulary are deliberately selected based on their relevance to the test scenarios (See Figure 1(a) ). However, such a setting may be infeasible in real-world applications especially when the prior information of the test scenarios is absent.To bridge the above gap, one straightforward idea is constructing a scenario-independent vocabulary during inference by exponentially enlarging the vocabulary size to cover all potential object categories that commonly appear across a wide scenarios. However, empirical evaluations in most existing studies are conducted with limited input vocabulary sizes (e.g., 17 categories in COCO [28] and 337 categories in LIVS [13] ). It is not clear whether state-of-the-art open-vocabulary detectors can still maintain their performance given exponentially enlarged vocabularies as input.In this paper, we put the above issue under the lens by introducing a new object detection task called large-and-open vocabulary detection. To simulate real-world scenarios, we extend the input vocabulary from dozens or hundreds to around two thousands during inference, encompassing frequently encountered object categories, while the training setup remains consistent with the conventional open-vocabulary detection (See Figure 1(b) ). Our experiments observe a substantial decline in detection accuracy of existing methods [10, 27, 45] as the size of the category vocabulary increases (See Figure 1 (c) as an example). This outcome aligns with expectations as the detection accuracy primarily hinges on visual-textural feature matching, and more categories in input vocabularies will inevitably lead to an increase in distractors, thereby escalating the complexity of feature matching and classification. The above drawbacks are suffered by most existing methods, which severely limits their application scenarios.To remedy the above drawbacks, we present the Large-and-Open Vocabulary Detector (LOVD), a new detection paradigm which is more scalable and robust to large input vocabularies, and can be seamlessly integrated with predominant detection methods to improve their open-vocabulary performance. One of our key contributions is the Image-to-Region Filtering (IRF) module. To alleviate the impact of category interference in large vocabularies, IRF performs image-level recognition first to build a more compact vocabulary out of the large one by filtering out a significant amount of distracting categories and identifying those highly relevant to the input scene. A more fine-grained region-level classification is then applied to each object proposal based on the compact vocabulary. By adopting the above coarse-to-fine philosophy, IRF can effectively tackle the challenge brought by large vocabularies and yields more accurate detection performance in an efficient manner. In addition, since the vision encoders of most VL models are pre-trained on the image level, directly transferring them to region-level classification may encounter significant generalization issues. In comparison, IRF combining both image and region-level recognition provides an alternative solution to ensure better generalization ability of pre-trained VL models.Most existing open-vocabulary detectors perform visual-textual feature matching in a global manner, i.e., each object region is represented by a global feature vector, which offers greater flexibility but may ignore detailed visual cues. In contrast, local approaches have also been explored recently in image recognition [50] , which align textual features with individual local image patches. While local matching excels at capturing granular details, it might not fully grasp the holistic context of the image. To harness the strengths of both global and local approaches, we propose the Cross-View Verification (CV 2 ) scheme to further enhance our IRF module, which performs image-to-region filtering in both global and local views, fostering an elegant cross-view interaction. Their predicted results are selected through a multi-branch voting mechanism to produce the final detection categories.In summary, the contribution of this paper is threefold. With its superior performance and adaptability, LOVD is wellpositioned to meet the demands of real-world detection tasks, paving the way for more intelligent and versatile vision understanding. Source code and pre-trained models will be released. 
related work 2.1 open vocabulary detection: 
 The open vocabulary detection approach, pioneered by OVR-CNN, uses captioning data to associate novel semantic categories with visual regions [49] . Capitalizing on the success of pre-trained VLMs, which merge extensive image and language vocabularies, subsequent research has harnessed these models for OVD enhancements [20, [24] [25] [26] 33] . The first to implement CLIP in this domain, ViLD introduced a method for instance-level visual-to-visual knowledge distillation [12] . Following this, the DETR-style OV-DETR model utilized VLMs to create adaptive queries [48] . Later models, including CORA [45] , incorporated region prompting [46] and anchor pre-matching to expedite training and better align disparate image data scales. Further strategies [5, 10, 22, 27, 31, 32, 37, 40, 42, 44, [51] [52] [53] to address data imbalances and feature misalignments have involved the use of more balanced datasets, pseudo labels, and expansive region-text pre-training. SIC-CADS [8] exploits global knowledge derived from CLIP to substantially refine existing OVD models. Our innovation, LOVD, specifically targets the refinement of image-to-region filtering to effectively manage the complexities of extensive vocabularies.Vision-Language Models (VLMs) have seen transformative developments with the advent of models like CLIP [33] , Align [20] , and COCA [47] , which harness contrastive learning to analyze expansive image-text datasets. These models are renowned for their ability to offer an in-depth, nuanced understanding of images within a comprehensive contextual framework, significantly surpassing traditional image analysis methods. VLMs play a critical role in diverse applications, from enhancing sophisticated image retrieval systems [1-3, 18, 21, 30] to developing innovative human-computer interaction techniques [34, 36, 39, 41, 43] . Innovations continue with models [4, 9, 11] , which extend the capabilities of VLMs in synthesizing and interpreting text-image content. Additionally, these advancements facilitate the integration of VLMs into dynamic environments where adaptive learning and context-aware processing are crucial.Matching Open Vocabulary Recognition, also known as zero-shot multi-label recognition, is a task where the objective is to categorize multiple tags relevant to an image. Although vision-language models (VLMs) are adept at recognizing a broad range of open-set categories in single-label scenarios [6] , their effectiveness diminishes in complex multi-label environments due to inadequate modality interactions. In contrast, models like RAM [50] , Tag2Text [19] , DualCoop [38] , and others [15] [16] [17] have shown significant advancements in managing tasks with extensive vocabularies by leveraging large-scale image-tag data for training VLMs. These models exhibit robust zero-shot learning capacities, particularly excelling in detailed region-specific semantics rather than just focusing on global image attributes. 
method 3.1 task definition: 
 Open-Vocabulary Detection. In open-vocabulary detection, a detector is trained using bounding box annotations and class labels of base categories 𝑉 𝐵 . During inference, given a pre-defined set of novel categories 𝑉 𝑁 , where 𝑉 𝑁 ∩ 𝑉 𝐵 = ∅, the detector aims to detect all objects belonging to an open vocabulary 𝑉 𝑂 = 𝑉 𝐵 ∪ 𝑉 𝑁 containing both base and novel categories. To this end, existing open-vocabulary detectors often operate in a two-stage manner, i.e., a category-agnostic localization step (e.g., using Region Proposal Network [35] ) followed by an open-vocabulary classifier. The text embeddings of category names extracted by a pre-trained VLM can be employed as the classification weights, and the open-vocabulary classification is then equivalent to visual-textural feature matching. Compared to its closed-set counterparts, open-vocabulary detection is able to recognize unseen categories in a zero-shot manner, which significantly benefits a wider range of application.Large-and-Open Vocabulary Detection. Though open vocabulary detection has largely advanced the field, existing studies [8, 44, 45, 53] are mostly conducted under an ideal setting, where the input vocabulary during inference is precise and meticulously crafted according to the input scenarios. This may be significantly diverged from real applications, as the prior information of testing scenarios can be insufficient to construct a compact and curated vocabulary. To circumvent this issue, a simple workaround is to extend the input vocabulary to include as many potential categories as possible. Therefore, it is natural and important to investigate the task of large-and-open vocabulary detection. Specifically, its training setting is the same as conventional open-vocabulary detection. During inference, the input vocabulary becomes exponentially larger and uncrafted, containing thousands of commonly encountered object categories, most of which do exist in the input image. Section 4.2 presents the detailed procedure on how to build such a large inference vocabulary for existing detection datasets. Large-and-open vocabulary detection has the potential to bridge the gap between real-world requirements and conventional evaluation settings, and is able to more comprehensively investigate open vocabulary detectors. In this section, we present the details of our proposed Large-and-Open Vocabulary Detector (LOVD) model. As shown in Figure 2 , it mainly consists of a category-agnostic localization module, a Image-to-Region Filtering (IRF) module, and a two-branch voting module. Given an input image, we first extract the feature map using the visual encoder of a pre-trained VLM. All potential objects are localized using an off-the-shelf region proposal network [35] , and their features are extracted with RoIAlign [14] from the image feature map. The IRF module then performs recognition for each object proposal based on the input vocabulary via a Cross-View Verification (CV 2 ) scheme. The final class label is determined by the two-branch voting module. In the following, we will elaborate on the implementation of each module.As the input vocabulary becomes exponentially larger, the category interference issue arises as a unique challenge of large-and-open vocabulary detection. Since object recognition needs to consider significantly more candidate categories with most of them acting as noisy distractors, the open-vocabulary recognition results will become less reliable. To mitigate this issue, we borrow a coarse-tofine philosophy and propose the Image-to-Region Filtering (IRF) module, which divides the open-vocabulary object recognition into two steps: image-level selection and region-level recognition. These two steps essentially involve two open-vocabulary classifiers with different purposes. Among them, image-level selection performs a one-pass recognition for the entire input image and aims to identify all existing object categories in the image. These potentially existing categories will be selected from the input large vocabulary (𝑉 𝐿 ) to constitute a pruned and more compact vocabulary (𝑉 𝐶 ) with a significantly reduced size. In the region-level recognition step, another classifier will take the new vocabulary as input to perform classification for each object proposals.Although conceptually simple, the IRF module significantly benefits large-and-open vocabulary detection from the following two aspects. Firstly, previous study [45] shows that it is non-trivial to transfer an image-level pre-traiend VLM to region-level classification. The IRF module with an image-level selection step can naturally alleviate the above generalization gap and effectively prune the input large vocabularies, ensuring more accurate final classification results. Secondly, the enlarged vocabulary size will linearly improve computational burden. Taking the cosine similarity computation between 𝑁 categories and 𝐾 object proposals as an example, the original compute count is 𝑁 × 𝐾. Thanks to the two-step coarse-to-fine pipeline, the cosine similarity for all the 𝑁 categories are only computed against the input image for one time in IRF, and the overall compute count becomes 𝑁 + 𝑀𝐾, where 𝑀 ≪ 𝑁 denotes the size of the pruned vocabulary. In our experiments with 𝑁 = 2000, 𝐾 = 1000, and 𝑀 = 15, IRF effectively reduces 99.15% of the overall computational complexity.Though with different purposes, the core components of the two steps in IRF module are both open-vocabulary classifiers. Therefore, how to design these classifiers are critical. As explained in Section 1, existing methods are restricted to either global or local representations, and may fail to capture detailed visual cues or lack a holistic understanding of the input scenes. We believe these two kinds of methods may largely cooperates with each other. To this end, we design two open-vocabulary classifiers, namely, a matching-based and a query-based method. The matching-based classifier represents the input visual content from a global view, while the query-based method focus on local visual cues.A Cross-View Verification (CV 2 ) scheme is further developed, which integrates the above two classifiers into the IRF module in a cooperative manner to further enhance the open-vocabulary recognition performance for large input vocabularies. As shown in Figure 3 , both the image-level selection and region-level recognition step contain a matching-based and a query-based classifier in parallel. In image-level selection, the two classifiers will produce two pruned vocabularies, which will then be respectively received by a different classifier in the region-level recognition step. More specifically, the vocabulary generated by the matchingbased (query-based) classifier in the image-level selection step will serve as input to the query-based (matching-based) classifier in the region-level recognition step. Finally, the IRF module will produce two separate classification results, which have been crossly verified by the two classifiers from both global and local views in different orders. As a consequence, we are able to achieve a better synergy between these two kinds of classifiers, giving rise to more reliable classification results.Matching-based Classifier. Similar to prior methods [45] , the matching-based classifier performs open-vocabulary recognition by matching the similarity of the visual input (i.e., either the input image or each object proposal) and the category names in the input vocabulary. It employs the pre-trained CLIP model [33] to extract visual and textual features, where the visual input is projected into a global feature vector 𝑓 and the vocabulary is mapped into the textual feature set {𝑡 𝑐 } 𝑁 𝑐=1 , where 𝑐 denotes the index category names in the vocabulary. Since the extracted visual-textual features have been aligned in a joint space, the classification results for the current visual input can be directly determined according to its cosine similarity score to each category name as follows:𝑚 𝑐 = exp (cos (𝑓 , 𝑡 𝑐 )) 𝑐 ′ ∈𝑉 𝑂 exp (cos (𝑓 , 𝑡 𝑐 ′ )).Query-based Classifier. To preserve fine-grained visual cues, the query-based classifier divides the visual input into 𝐻 × 𝑊 patches. A vision Transformer encoder is employed to convert the input patches into visual feature 𝐹 ∈ R 𝐻𝑊 ×𝑑 , with 𝑑 indicating the feature dimension. Meanwhile, the category names in the vocabulary are also encoded into textual features {𝑡 𝑐 } 𝑁 𝑐=1 using pre-trained VLM. A lightweight Transformer decoder is further adopted, which takes the textual features as queries and performs cross-attention between textual and visual features to generate a set of textual embeddings {𝑒 𝑐 } 𝑁 𝑐=1 . Finally, each textual embedding is passed through a linear layer followed by a Sigmoid activation to predict the probability score 𝑞 𝑐 for the corresponding category. Training. We adopt an off-the-shelf category-agnostic detector for object localization. The visual and textual encoders are pretrained through contrastive learning following [33] . The matchingbased classifier is based on these two encoders and does not require any further fine-tuning on object detection data. The query-based classifier is trained on both image-level and region-level classification. For image-level training, we follow [50] to collect image-level pseudo labels. For region-level training, groundtruth bounding box annotations are directly used as training labels. Training is conducted by optimizing the following binary cross-entropy loss:EQUATIONwhere 𝑞 𝑐 denotes probability score of the 𝑐-th category predicted by the query-based classifier, and the groundtruth label 𝑦 𝑐 = 1 indicates that the 𝑐-th category exists in the input image and 𝑦 𝑐 = 0 otherwise. Since the IRF module with the CV 2 scheme is compatible with most predominant open-vocabulary detectors, it can be easily applied to and significantly benefits existing methods as shown in our experiments.Inference. For each input image, the matching-based and querybased classifier in the image-level selection step will first produce the refined vocabulary 𝑉 𝑀 and 𝑉 𝑄 , which contain the top 𝐾 categories with the highest scores predicted by the two classifiers, respectively. In the region-level recognition step, for each object proposal, the two classifiers will separately predict a probability score for each of the categories in the refined vocabularies. To obtain the final classification results, we design a two-branch voting mechanism as follows.EQUATIONwhere 𝑚 𝑐 and 𝑞 𝑐 are probability scores for the 𝑐-th category predicted by the matching-based and query-based classifiers, respectively. The hyperparameters 𝛼 and 𝛽 are used to balance the confidence of the two classifiers. The final classification result is determined as the object category with the highest class score 𝑝 𝑐 . 
experiments: 
 In this section, we extensively evaluate the proposed LOVD method across various open-vocabulary and large-and-open vocabulary detection tasks. Details about the datasets and evaluation metrics are presented in Section 4.1, and the specific implementation nuances of our LOVD are outlined in Section 4.2. We perform comparisons with leading methods in Section 4.3, highlighting the superior capabilities of our approach. Further, we conduct the ablation study to investigate the IRF module, the CV 2 scheme, and hyperparameters in Sections 4.4.We conduct experiments on the two popular benchmark datasets, including COCO [28] dataset and LVIS [13] dataset. To verify the robustness of the compared detectors, both the open-vocabulary and large-and-open vocabulary settings are adopted for the two datasets. More detailed explanations are as follows.Open-vocabulary Setting. Following the conventional setting [3] , the vocabulary of COCO dataset is partitioned into 48 base categories for training, and 17 novel categories only for testing the open-vocabulary ability of models. Similarly, the LVIS dataset includes 1,203 categories, classified as frequent, common, and rare categories based on their occurrence. Following prior methods [12] , the frequent and common categories are used for training, and the rest 337 rare categories are adopted as novel categories for testing. Evaluation Metrics. Following the prior methods [45, 53] , we utilize the 𝐴𝑃50 of bounding boxes as the evaluation metric for the COCO dataset. This metric calculates the average precision at an intersection over union (IoU) threshold of 50% for each category and then computes the overall average across all categories. In the open-vocabulary setting, we specifically compute the 𝐴𝑃50 for the novel categories. In the open-and-large vocabulary setting, we assess the performance across novel, base, and all categories.For the LVIS dataset, we compute the 𝑚𝐴𝑃 of masks averaged on IoUs from 0.5 to 0.95. Similar to the COCO dataset, we compute the 𝑚𝐴𝑃 for the novel categories for the open-vocabulary setting, while perform evaluation for rare (novel), common, and frequent categories in the open-and-large vocabulary setting.In addition, We are also interested in comparing the performance degradation on the novel categories of detectors from open vocabulary to large-and-open vocabulary settings. Therefore, a relative metric named the Decay Ratio (DR) is computed as follows:EQUATIONwhere LOV-AP novel and LOV-AP novel denote the AP value for novel categories under the open-vocabulary and large-and-open vocabulary settings, respectively.Cross-dataset Evaluation . Following previous work [10, 27, 40, 49] , we train our LOVD on the training set of COCO and conduct zero-shot transfer experiment on PASCAL VOC [7] to evaluate the generalization ability of the proposed method, using AP50 as the metric.Building Large Vocabulary. The number of object categories in both COCO and LVIS is limited for large-and-open vocabulary evaluation. To augment the existing vocabulary for these datasets, we apply two principled criteria: (1) The newly added categories must not overlap or be synonymous with any categories of vocabulary in the validation set. (2) They should be absent from all images in the validation set. Our selection starts by gathering more than 10,000 common categories from databases like OpenImages [23] and ImageNet-21K [6] as the initial list. Then, we utilize the multi-modal large language model [29] to eliminate categories from the initial list that already exist in the validation set, and employ CILP [33] to filter out the semantically redundant categories. Finally, we elaborately refine the categories manually to build the large vocabulary with 2,000 categories in total. For more details, refer to the supplementary materials.Training Setups. We train the Query-based classifier on COCO with the supervision of pseudo tag labels generated by [50] with the binary cross entropy loss. To avoid label leakage, tags associated with the 17 novel categories are excluded from the pseudo-labels during training. Similarly, the training protocol on the LVIS dataset follows this principle that excludes categories related to rare categories. We train the query-based classifier on an NVIDIA Tesla A100 using PyTorch, employing the AdamW optimizer with an initial learning rate of 2 × 10 -5 . The batch size is set to 4, and the training process runs for a total of 10 epochs. Based on the observations in our ablation studies (as shown in n Section 4.4), the value of hyperparameters 𝛼 and 𝛽 in (3) are set to 1.0 and 4.0, respectively. The number of selected categories 𝐾 in image-level selection step is set to 15. We keep the above setting consistent for both COCO and LVIS datasets. Since the main contribution of our LOVD is the IRF module with the CV 2 scheme, the other modules are compatible with most existing detectors. Therefore, to evaluate the effectiveness and adaptability of our LOVD, we integrate it with different prior open-vocabualry detectors that employ different network architectures and learning principles. These include prompt learning (CORA [45] ), weakly supervision (Detic [53] ), and region-text alignment (BARON [44] , VLDet [27] ). We present the comparative results of the original open-vocabulary models and their LOVD-enhanced counterparts (denoted with the postfix "+LOVD" in Table 1 -3).Results on COCO. Table 1 reports the quantitative results of the compared methods and the integration of LOVD with these models on the COCO benchmark dataset. All the state-of-the-art open-vocabulary models, including those trained with extensive vocabularies of 21K categories from ImageNet-21K [6] such as Detic [53] and Baron [44] , show a significant decrease in performance under the large-and-open vocabulary setting. In contrast, integrated with the proposed LOVD, it consistently and significantly enhances the performance of all representative models in identifying novel categories across all the inference settings. Specifically, for the openvocabulary setting, it shows that the integration of LOVD achieves an improvement of 5.0 in terms of 𝐴𝑃50 for the Detic method [53] . For the large-and-open vocabulary setting, the performance improvement is more significant for the novel categories. It achieves a maximum improvement of 8.8 in terms of 𝐴𝑃50 compared with the original methods, and establishes a new SOTA performance when incorporated into CORA. Moreover, LOVD exhibits great robustness towards large vocabulary in terms of decay ratio. With the assistance of LOVD, all representative methods achieve substantial improvements, with the smallest observed decline in performance being only 1.5%. These above results show that our proposed LOVD can effectively mitigate the category interference issue for the openand-large vocabulary task.Results on LVIS. LOVD demonstrates a similar superiority on the LVIS benchmarks, as detailed in Table 2 . It shows that when expanding the categories of the input vocabulary, the existing leading methods VLDET [27] and Detic [53] Table 3 : Zero-shot transfer results on the Pascal VOC. We report AP50 as the evaluation metric.the 𝑚𝐴𝑃 scores for the novel categories by more than 2.0 across these base models.Cross-dataset Generalization . In the cross-dataset generalization experiment, we further conduct evaluation on the Pascal VOC dataset for the models trained on the COCO dataset. As shown in Table 3 , integrated with our LOVD, it significantly improves the AP50 scores by 2.4 and 0.7 in both open-vocabulary and large-andopen vocabulary settings for the CORA method, and also improves the DR scores by 2.7. These results illustrate our remarkable enhancement in generalization ability across different datasets.We perform ablation studies to evaluate the key components of the proposed method. All the evaluations are conducted on the COCO dataset. "LOV-AP novel " in Table 4 -6 denotes the AP50 score for the novel categories under the large-and-open vocabulary setting.Image-to-Region Filtering. We evaluate the performance of various IRF variants on top of the baseline method CORA [45] to showcase the significance of our IRF, as shown in Table 4 . In the large vocabulary setting, the method proposed by SIC-CADS [8] , which multiplies region-level and image-level scores, demonstrates inferior performance compared to the method of filtering before classification. Our IRF module, integrating image-level selection with region-level classification, achieved the highest performance. Table 4 also shows the results of varying values of 𝑘 in Top-k operation of image-level recognition. We observe that in the scenes of the COCO dataset, typically 10-20 object categories appear, so select 𝑘 from the range of 10-20. When using a smaller 𝑘, too many ground truth categories are filtered out. However, if a large 𝑘 is used, too many interfering categories are fed into the region classifier, resulting in a significant impact from the large vocabulary. Since Cross-view Verification . As shown in Table 5 , Different variants of CV 2 can be employed in our LOVD approach. We evaluate the performance of CV 2 variants on top of the baseline module the region-level Matching-based Classifier to showcase the significance of our CV 2 . By integrating a image-level Query-based Classifier, we establish the first branch, which notably increases AP from 27.6 to 35.0. Similarly, we evaluated a Region-level Query-based Classifier as the baseline, and by adding an image-level Matching-based Classifier, we formed the second branch, increasing AP from 12.4 to 13.0. However, these approaches result in diminished performance, indicating that the individual branches alone do not yield optimal results. Merely combining the results from both branches yields an AP of only 35.3, indicating that the branches alone do not provide optimal outcomes. It is the combination of Two-branch Voting that achieves superior performance, with an AP of 36.4, underscoring the importance of synergistic integration. Furthermore, we evaluate configurations that omit the cross-view combination, which leads to a marked decrease in performance. This decline is attributed to error accumulation among identical models, resulting in compounded inaccuracies. These findings suggest that simply amalgamating modules without the strategic integration provided by CV 2 does not effectively leverage their complementary capabilities and may even be counterproductive. considers present while another branch deems absent. The higher the coefficients, the greater the penalty applied to the scores. If the coefficients are set too high, the scores of the correct categories may become excessively low. Conversely, if the coefficients are too low, the scores for incorrect categories might be too high. Optimal values of 𝛼 and 𝛽 effectively reduce the likelihood of false negatives and false positives. We adopt the best-performing hyperparameters 𝛼=1.0 and 𝛽=4.0 from the COCO experiments. Although the ablation studies are conducted on the COCO dataset, given the similar scale of scores between COCO and LVIS datasets, we apply 𝛼=1.2 and 𝛽=4.0 as the hyperparameters for the LVIS dataset.Scale of Vocabulary. We investigate the effect of introducing a variable size of large vocabulary on model performance, as depicted in Figure 1 (c) (See the supplementary materials for more details). When a modest number of 500 extra categories are introduced, OV models experience a notable decrease in accuracy. As we incrementally increase the count of extra categories from 0 to 2000, the performance of OVD methods consistently worsens, while our LOVD remains largely unaffected. This evidence indicates that our approach possesses a superior adaptability in facing real-world scenarios, demonstrating resilience against the influx of distracting categories. 
conclusion: 
 In this paper, we introduce a novel task of large-and-open vocabulary detection, crucial for enhancing the real-world applicability of open-vocabulary object detection systems. To effectively manage the challenges posed by vast input vocabularies, we propose a new open-vocabulary detection method LOVD. By incorporating the Image-to-Region Filtering (IRF) module and the Cross-View Verification (CV 2 ) scheme, LOVD significantly improves detection accuracy and robustness against the interference of large vocabularies. Our extensive experiments on various datasets underscore LOVD's distinct advantage over existing approaches, showcasing its enhanced scalability and robustness in handling large vocabularies.In the future, we will further expand the scale of the vocabulary to better simulate real scenarios under the large-and-open vocabulary detection setting. In addition, we will also experiment with more advanced VLM and network architectures to improve the performance and adaptation our of method.  
 
 Appendix:  


Decision: There is a consensus to accept this paper. The author's responses to the reviewers' comments didn't completely satisfy the reviewers.
In particular, there is still concern about both the effectiveness of the proposed method and its efficiency (FPS).
I suggest accepting this paper, given the general consensus.
However, I ask the authors to add some of the valuable Q&A to the paper or the supplementary for making new readers apart from reviewers available with these details.

Reviews: review: paper_summary: paper_summary: This work introduces a new setting called Large-and-Open Vocabulary object Detection (LOVD), which tests object detectors with extensive vocabularies containing thousands of unseen categories. To address the challenges posed by this setting, the authors propose a Large and Open Vocabulary Detector (LOVD) featuring an Image-to-Region Filtering (IRF) module and a Cross-View Verification (CV2) scheme. These components work together to filter and classify relevant categories, making the detection process more scalable and robust in real-world scenarios.
strength_weakness: strength_weakness: 1. The motivation for the paper is interesting and convincing
2. The completeness of the paper is very high, the writing is clear, and the experimental results are adequate
limitations: limitations: 1. (Major) The choice of K in top-k selection seems to be sensitive to the final performance (Table 4). Especially, COCO and LVIS share the same image sources from COCO, the optimal selection of K introduced in this paper for them is 15 which may not fit the other datasets. In this case, how to handle the selection of K across different datasets, please verify.  (Minor) Additionally, is it possible to adaptively choose K?
2. (Minor) The method for constructing the Large-and-Open Vocabulary Setting may be inappropriate. The paper filters similar texts based on text feature similarity (supp. Lines 15-16) to ensure category diversity. However, using text similarity alone to determine semantic similarity and filter duplicate categories may be problematic. For example, using CLIP, the text feature cosine similarity between 'pool table' and 'dinner table' is 0.73, yet they are visually very different. (Major) However, more importantly, LOV appears to be constructed separately for different datasets (Lines 578-580). Therefore, the results obtained using different LOVs for different datasets may not be reliable. Could you provide a unified LOV set for various datasets (at least for COCO and LVIS)?
3. (Major) The proposed Image-to-Region Filtering method reduces the vocabulary from VL to VC by determining the potential categories present in the image (Lines 378-380). There are two concerns: 1) Fairness of comparison. The construction of LOV has already excluded highly similar texts, so Image-level selection can easily exclude newly added extra vocabulary (VL -> VC), making the proposed method's setting on LOV close to or even weaker than the OV setting ("typically 10-20 object categories appear", Lines 805-808) whereas other models handle the vocabulary size of VL, making it difficult to evaluate the superiority of the proposed methods. 2) The performance improvement for image-level selection shown in Table 4 is significant, likely due to the vocabulary filtering making the model easier to recognize. Please provide more analysis for validation.
4. (Minor) Inconsistent naming, VLDET in Table 2 and VLDet in Table 1.
paper_summary: paper_summary: The paper tackles open-vocabulary object detection (OVD) for large (more than 1k) vocabularies, in which standard solutions for OVD have degraded performance due to the high number of distractor categories. The authors propose a two-stage, two-branch module designed first to filter out most of the improbable classes using global image-level information and then reuse those modules on regions with the new reduced vocabulary.
strength_weakness: strength_weakness: - The tackled problem of large-scale open-vocabulary detection is interesting and relevant. 
- The authors covered the relevant literature well. 
- Experimental evaluation is extensive and shows improvements over SotA on OVD tasks both with and without enlarged vocabularies. The authors also performed ablation studies for most of the design choices.
limitations: limitations: - There are some omissions in the presentation and motivation of the task and the proposed solutions that could be improved, such as the contextualization of the task (what are the possible applications of LOVD? Do domain-specific applications often need smaller, specialized vocabularies, conversely?) and some design choices (why using two different open-vocabulary modules in IRF and in that configuration? How sensitive is the proposal to choosing the top categories K, \alpha, and \beta? How can we choose them for datasets other than the tested ones? ) 

- It seems the method is somewhat inspired by SIC-CADS [8] in the image-to-region filtering rationale, and thus I expected additional comparisons. Only Detic + SIC-CADS is considered, while in [8], results are available for other models in which also the proposed approach has been tested (e.g., BARON, CORA RN50x4) 

- Have the authors recomputed SotA models to obtain numbers in Tables 1, 2, and 3, or have they reported numbers from other papers? I see small discrepancies (e.g., Detic + SI-CADS 26.2 (paper) vs. 26.5 [8]) that, if due to slight implementation differences, may hinder the significance of some small performance improvements reported. 

Minor issues/typos: 
- L99: backend --> backed 
- L392: improve --> increase 
- L399-400: what do the authors mean by overall computation complexity? Are they including backbone models in the 99.15% reduction? 
- L405: are critical --> is critical  
- L629: fix double LOV-AP 
- L674: CILP --> CLIP
paper_summary: paper_summary: This work aims to tackle a new problem, Large-and-Open Vocabulary Object Detection, in which thousands of unseen categories exist in the testing vocabulary. To address the challenge, this work proposes a coarse-to-fine method by first filtering some unrelated categories and building a compact vocabulary and then using the new image-specific vocabulary as the the classifier. Experiment results show improvements on both traditional open-vocabulary setting and Large-and-Open Vocabulary Object Detection setting.
strength_weakness: strength_weakness: 1. This work tackles a new and challenging problem.
2. The proposed coarse-to-fine method and Cross-View Verification scheme can alleviate the classification difficulty.
3. The proposed method is general and can be applied to many existing open-vocabulary methods.
limitations: limitations: 1. Figure1(c) illustrates the impact of the number of unseen categories for different open-vocabulary detectors during testing. However, it seems that the models are trained on OV-COCO. I wonder whether the same phenomenon can be found when the detector is trained on OV-LVIS with much more categories, especially detectors trained with federated loss. Since federated loss samples a small vocabulary during training while using all classes in the testing, which is similar to the Large-and-Open Vocabulary Object Detection setting.
2. In Table 4, the best k is 15. As the number of categories for some images in LVIS is possibly larger than 15, please provide the recall rate for each k, especially on LVIS dataset.
3. It seems that the proposed classifier needs many complex operations. Please compare the fps with the baseline.
4. The results in Table 5 is poor-organized and confusing. More experiment details should be provided.
5. In region-level recognition, different vocabularies are used in the matching-based and query-based classifier. I want to know the performance using the same vocabulary (the intersection or union of two vocabularies).
paper_summary: paper_summary: The number of unseen categories in the open-vocabulary object detection task is now normally limited. The performance of existing methods degrades when the number of unseen categories increases, especially when the number grows to extremely large. To tackle this problem, this paper formulates a new task named Large-and-Open Vocabulary Object Detection. Along with this new setting, a new detector named LOVD is proposed, which contains an Image-to-Region Filtering (IRF) module and the Cross-View Verification (CV2) scheme.
strength_weakness: strength_weakness: 1.	The problem that this paper is trying to solve is important and necessary, which is the trend for future work.
2.	The newly proposed method LOVD is simple yet effective, achieving good performance compared with existing state-of-the-art methods.
3.	The parameter of LOVD is not very large, bringing little computational cost when inserting into existing methods.
4.	Authors claim that the source code will be made publicly available. Hoping the generated dataset on COCO and LVIS can also be public to serve as a good baseline for the research community.
limitations: limitations: 1.	How to decide whether the number of unseen categories is large enough? What is the standard to distinguish between normal and large regarding open vocabulary object detection?
2.	Although the amount of parameters that LOVD brings is not very large, it is recommended to show the comparison of inference speed and training time to reflect the computational cost under real scenarios.
3.	The future work and limitations analysis are missing. This work, as the first paper for the new setting, is recommended to give more insight into future work and some analysis of the limitations of LOVD.
4.	The used template is not the latest one.