VL-SAM-V2: Open-World Object Detection with General and Specific Query Fusion
Authors: Zhiwei Lin
Yongtao Wang
Venue: NeurIPS 2025 poster ()
Keywords: open-world
open-vocabulary
open-ended
object detection
Primary area: applications

Abstract: Current perception models have achieved remarkable success by leveraging large-scale labeled datasets, but still face challenges in open-world environments with novel objects. To address this limitation, researchers introduce open-set perception models to detect or segment arbitrary test-time user-input categories. However, open-set models rely on human involvement to provide predefined object categories as input during inference. More recently, researchers have framed a more realistic and challenging task known as open-ended perception that aims to discover unseen objects without requiring any category-level input from humans at inference time. Nevertheless, open-ended models suffer from low performance compared to open-set models. In this paper, we present VL-SAM-V2, an open-world object detection framework that is capable of discovering unseen objects while achieving favorable performance. To achieve this, we combine queries from open-set and open-ended models and propose a general and specific query fusion module to allow different queries to interact. By adjusting queries from open-set models, we enable VL-SAM-V2 to be evaluated in the open-set or open-ended mode. In addition, to learn more diverse queries, we introduce ranked learnable queries to match queries with proposals from open-ended models by sorting. Moreover, we design a denoising point training strategy to facilitate the training process. Experimental results on LVIS show that our method surpasses the previous open-set and open-ended methods, especially on rare objects.

VL-SAM-V2: Open-World Object Detection with General and Specific Query Fusion
Zhiwei Lin Yongtao Wang"
Wangxuan Institute of Computer Technology, Peking University, China {zulin, wyth&pkk.. eesb@ppu.edu.coo edu.
Abstract
Current perception models have achieved remarkable success by leveraging large- scale labeled datasets, but still face challenges in open- open-world world environments with novel objects. To address this limitation, researchers introduce open-set perception models to detect or segment arbitrary test-time user-input categories. However, open-set models rely human involvement provide predefined object categories as input during inference. More recently, researchers have framed a more realistic and challenging task known open-ended perception that aims discover unseen objects without requiring any category-level input from humans at inference time. Nevertheless, open-ended models suffer from low performance compared to open- set models. In this paper, we present VL-SAM-V2. an open-world object detection framework that is capable of discovering unseen objects while achieving favorable performance. To achieve this, we combine queries from open-set and open-ended models and propose general and specific query fusion module to allow different queries to interact. By adjusting queries from open-set models, we enable VL- SAM-V2 V2 to be evaluated in the open-set or open-ended mode. In addition, learn more diverse queries, we introduce ranked learnable queries match queries with proposals from open-ended models by sorting. Moreover, we design denoising point training strategy to facilitate the training process. Experimental results on LVIS show that our method surpasses the previous open-set and open-ended methods, especially rare objects.
Introduction
Deep learning has made significant breakthroughs in computer vision tasks, enabling substantial progress in object detection tasks. Traditional object detection approaches [34 16] typically employ closed-set paradigm, in which detection models are restricted to recogmizing and locating objects that are seen the training set. However, closed-set approaches have notable limitations, particularly in real-world scenarios where novel or unseen objects appear. For instance, in an autonomous driving system, if its object detector trained to identify vehicles and pedestrians encounters novel object (22). wild animal, it may either misclassify the object fail to detect it, which can even lead to serious traffic accidents.
such drone
To address this challenge, open-world methods 12 be more flexible by enabling them handle novel and unseen objects. Open-world world methods generally can be divided into two categories: open-set (or open-vocabuse veee-vocabulrr)) [14] [9] and open- ended 25 27). Open-set methods can identify objects within predefined object category list, which is provided by humans during the inference phase, even if unseen objects are present in the list. However, these open-set models cannot predict objects outside the predefined object category list, which limits their ability to discover new object categories automatically. For instance, the
have been proposed, allowing models
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

VL-SAM
Vision-language Model
General Queries
Segmentation mask
mask to box
Boundling Box
SAM
VL-SAM-V (Our)
Vision- language Model
General Queries
SAM (optional)
Segmentation
General and Specific Query Fusion
mask
Open-set Model
Specific Queries
Box Head
Bounding Box
Figure 1: Illustration of VL-SAM-V2. VL-SAM-V2 combines the general queries from VL-SAM and the specific queries of an open-set model with query fusion module.
predefined object category list only contains vehicles and pedestrians, the open-set models still cannot detect drones or wild animals in the input images. Therefore, they need humans to intervene by providing more comprehensive list including these objects. Additionally, although open-set models perform well on frequent objects, they often struggle with rare objects [29]. In contrast, open-ended models aim to identify and categorize all objects given images without requiring a predefined object category list provided by humans [25]. This capability is usually achieved by incorporating large vision-languee especially when dealing with crowded objects, which typically appear in frequent object categories.
models. However, open-ended models suffer from low performance [43 27].
To this end, we introduce VL-SAM-VE an open- world object detection framework that combines the strengths of open-set and open-ended methods to achieve favorable performance in both rare and frequent object categories while equipped with the ability discover novel objects, as shown in Figure I Specifically, we treat object queries from open-set models as specific queries, since they specialize in frequent objects. Meanwhile, we adopt the VI.-SAM pipeline to generate object queries in open-ended manner and view them as general queries, because of the generalization of large vision-language models. Then, we propose general and specific query fusion module to fuse two distinct types of queries with the attention mechanism in lightweight manner. Since the VL-SAM only provides point prompts, we present ranked learnable queries by sorting the general point prompts based their scores and matching them with the learnable queries to compose general queries. In addition, to facilitate the training process, we introduce the denoising points strategy by adding both positive and negative points the noisy general queries. This denoising process helps models to learn to distinguish positive and high-accuracy general queries and to filter out negative queries point prompts from VL-SAM. During inference, VL-SAM-V. V2 can be dynamically adapted as an open-set or open-ended model based on the predefined object category list. When the list is empty, VL-SAM-V2 is viewed
open-ended model. Experimental results show that VL-SAMMV2 significantly improves detection performance on rare objects compared to existing open-set models by discovering novel objects with general queries. Meanwhile, VL-SAM-V2 compensates for the poor performance of open-ended models.
The main contributions of this work summarized as follows:
We introduce an open-world object detection framework, VL-SAM-V2, to combine the strengths of open-set and open-ended methods with the general and specific query fusion module. VL-SAM- V2 can dynamically adapted as an open-set or open-ended model.
We design ranked learnable queries to transform point prompts from VL-SAM into object queries and present denoising points strategy to facilitate the training process.
Experimental results show that VL-SAM-V2 achieves both state-of-the-a art open-set and open-ended object detection performance on the LVIS dataset a zero-shot manner.
Related work
2.1
Open-set Object Detection
Open-set or upen-vocabulary object detection methods aim detect objects predefined object category list given by humans (29). To achieve this, current open-set object detection methods

---

alignment task, allowing detectors to match unseen object categories with visual information through the alignment process. With the advent of contrastive learning 15 3. vision-language alignment models (e.g., CLIP (3) achieve favorable zero-shot alignment performance for unseen objects. Based on this, many open-set object detection methods use a proposal network to obtain foreground object bounding boxes and embeddings, and then use CLIP as the open-set classification module to predict their categories. However, CLIP is trained to align the whole image and texts, while the open-set object detection task favors region-leve level feature alignment.
often transform the open-set detection task into a
To address this, GLIP [24] proposes an object-aware vision- rision-language region-level alignment task with phrase grounding to pre-train open- world object detectors. DetCLIP [40] separates the texts in GLIP into a parallel concept and regards each object category as an individual input. GLIPv2 24 further improves on GL.IP by introducing deep fusion block for better vision-language fusion and intra- and inter-image-text contrastive losses. GrounddiggDINO (29) follows the idea of GL.IP to present image-text cross-modality fusion, and proposes object queries by text information. OWL 32
scales the training dataset to 10M examples with pscudo-annotation and a self-training strategy. FOLO-World 7 adopts the efficient YOLO-series backbone to achieve real-time open-set object detection. More recently, due to the emergence of large language models and vision-language models, many open-set methods try to adopt them for additional supervision. DetCLIPv3 41 combines the object-level and image-level generation tasks with the open-set object detection task. LLMDet (IT) explores two-stage co- co-training by generating image-level and region-leve detail captions with a pre-trained uisionllangaae model. In addition, some works (99 43 directly fine-tune pre- pre-trained
vision-language model to address the open-set object detection by following pix2seq 4
However, the above methods require predefined object category list given by humans and cannot discover novel objects by themselves. In this paper, we propose prompting the open-set method using point priority from large vision-languag models and enabling the proposed model to discover novel objects not the predefined object category list.
2.2 Open-ended Object Detection
Open-ended object detection methods can directly predict all objects in given images without any predefined object category list from humans. GenerateU [25] first introduces the open-ended problem and proposes generative framework with a large language model to generate object categories from object queries. In addition, it constructs a large dataset with pseudo bounding boxes and caption pairs to fine-tune the whole model. To alleviate the training costs, VI-SAM (77 presents a training-fres open-ended object detection and segmentation framework that connects large vision- vision-language model and the segment- anttinn model (SAM) (88) with attention maps as the intermediate prompts. Moreover, some large vision- -siounlanguaae models 199 43 20 introduce specific tokens, like '<Det>, to allow models to predict objects in an open-ended manner. Though good grounding performance of these vision-language models is achieved, the accuracy of detecting all objects is very low.
Though some progress has been made on the open-ended detection task, the performance of current open-ended methods is significantly lower than that of open-set methods. To address this issue, we propose a general and specific fusion pipeline by fusing the queries from open-set and open-ended models. The proposed pipeline preserves the strengths of open-set methods' high performance and can discover novel objects like open-ended methods.
3 Method
3.1 Preliminary of VL-SAM
VL-SAM 27 is a training-free open-ended object detection and segmentation framework that connects vision-languat and segment-anythie models with attention maps as the intermediate prompts. utilizes vision-language model to generate the categories of all objects in the given image and stores the corresponding attention maps. Then, VL-SAM adopts the head aggregation and attention flow mechanism to aggregate and propagate attention maps through all heads and layers. After that, sampling strategy is applied to the refined attention maps to generate positive and negative points as the point prompts for SAM. In addition, VL-SAM uses several ensemble strategies to obtain more comprehensive object points.
3

---

SAM (optional)
General
Vision-
Queries
General and Specific Query Fusion
language
Model
Query Categories
Box Predictions
Open-set Model
Predefined Object Category
Spocific Querics
Box Hoad
List (optional)
The overall pipeline of VL-SAM-V2. VL-SAM-V2 utilizes a vision-language model to generate general queries and a standard open-set set detection model to generate specific queries. Then, the two distinct queries are sent to the general and specific query fusion module for interaction. Finally, a box head and an optional SAM are applied to predict the perception results. During the training, we only fine-tune the general and specific query fusion module and the box head. In addition, by controlling the predefined object category list, VI.-SAM-V2 can operate open-ended mode.
Figure 2:
In this paper, we adopt VL-SAM to generate point prompts as proposals for the general queries in an open-ended manner.
3.2 VL-SAMM-2
As shown in Figure we provide an overview of the proposed framework. Specifically, given an image input, first use VL-SAM to discover all objects in the image and sample their point coordinates according to the attention maps. Then, we combine the discovered objects with the predefined object to form the final object set for the open-set model to generate specific queries. Meanwhile, the sampled points are converted into initial bounding boxes and matched with learnable queries compose general queries. After that, the general and specific queries sent the general and specific query fusion module to update the queries. Finally, the box head and optional SAM are applied to predict the final results.
Denoising Points. VL-SAM-V2 employs VL--AMM to generate point prompts to compose general queries. However, during training, the inference of VI-SAM will incur additional training costs due to the use large vision-language model. In addition, directly training with a few points from VL-SAM can lead to unstable training and poor convergence. Inspired by the denoising anchor boxes trick in the training of DETR [2), we propose denoising points to address this issue. Specifically, we randomly sample positive points the ground-truth bounding boxes and negative points outside the boxes to replace the point proposals generated by VL-SAM. The sampled noisy points are sent to the decoder, and the models are trained denoise the noisy points into corresponding bounding boxes and labels. More concretely, we follow DINO [42 to sample points by adding coordinate noise A.X and AY with two parametrrss A₃ and A2:
Axx < A1 X w/2 < AX,I < X2 w/2, AY < / < AY, < X2 h/2,
where p and denote positive and negative points, and h indicate the width and height of the ground-truth bounding box for denoising. The A.X and AY are sampled from the uniform distribution according to the range in Eq.
To convert sampled noisy points into bounding boxes and general queries for subsequent fusion and decoding, we utilize multi-level image features to predict the initial bounding boxes according to the coordinates of noisy points and introduce ranked learnable queries. Specifically, given the coordinates of a noisy point (Xnoiee; Y Ynoise) and the multi-level image features f. we interpolate the image

---

fupdate
ABg
General Boxes Bg
General and Specific Query Fusion
Query-Lo-Tedt
Query toy-tto-lmnn lo-lmage
Box Head
General
clles-attention
Pross-attention
Queries Qp
self-atlenttion
Text Feature
Shared Weights
Image Feature
Sharec Weights
Unshared
Specific
Queries Qs
Query-to- Text
Query-to Image
Box Head
Specific
cetssaatenntinn
boss-attention
Boxes B
AB; | update
Figure 3: Illustration of general and specific query fusion module. General and specific queries interact with a self-attention mechanism. Then, the shared query-to-text and query-to-image cross- attention applied for the queries independently. Finally, the unshared box heads predict the offset of corresponding bounding boxes. During the training, we only update the parameters in the self-attention and box heads.
features in (Xnoises Yoosse) for all levels and obtain sampled point feature fx noiss Pooisgg.noiis noise" where I denotes the 1-th level of the image features. After that, we send the point feature f [' Yuniee to linear layer and layer normalization, followed by box head in the open-set model to obtain the initial bounding box. The initial bounding box for each point is combined with the ranked learnable query to compose general queries and sent to the fusion module and box head.
Ranked Learnable Queries. Each point prompt noisy point should be matched with learnable query for subsequent fusion and decoding. However, point prompts or noisy points are sampled randomly without any order. A simple way is to set the same learnable query for each point, without considering their difference in coordinates or features. Intuitively, different queries are expected to handle different objects. Therefore, we introduce ranked learnable queries by matching different points with different queries. Specifically, for noisy points, we send f head obtain the score of each noisy point. Then, rank the noisy points from largest to smallest based on their scores. Meanwhile, we set N additional learnable queries and fix their order after initialization. Finally, the ranked noisy points matched with the learnable queries by For example, the noisy point with the largest score matches the first learnable query. Since the number of noisy points may not match N, simply discard the redundant noisy points ranked learnable queries.
moijge: Vegige. to the classification
Similarly, during inference, the points sampled by VL-SAM can be ranked by scores and matched with ranked learnable queries.
General and Specific Query Fusion. As shown in Figure 3 after obtaining the initial bounding box and ranked learnable query of each point, we combine them with the bounding boxes and queries generated by the open-set model and send them into the general and specific query fusion module, which is distributed each transformer decoder layer the open-set model. The boxes and queries of sampled points are treated general boxes B, and queries Q, since they are prompted by the large language model, while the boxes and queries from the open-set model are denoted as specific boxes B, and queries Q.. Since the number of sampled points varies, we adopt the self-attention module for the interaction between the general and specific queries, as shown in Figure Specifically, Qg and Qe are first concatenated and sent to a standard self-attention module to obtain the fused queries. Then, the fused queries are separated and sent to a shared query-to-text cross-atd cooss-attntionn, a shared query-to-image
moss-attention, and a shared FFN to obtain updated queries Q2 and Q.. After that, two unshared box heads applied to Q, and Q. to predict the coordinate offsets of bounding boxes A B, and A B,, respectively. Finally, we update the B, and B, as follows:
By By + AB B, - B, + AB.
(2)
5

---

During training, we only update the parameters in the self celf-attention and box heads.
Loss Function. We adopt the grounding losses Lorounding and the generation losses ggeneratinn in LLMDet [II] as the supervision to fine-tune VL-SAM-V2. The grounding losses contain vision- language alignment and bounding box regression losses, while the generation losses comprise region-level and image-level caption losses. In VL-SAM-V2, the alignment process between ground- truth and general/specific boxes is performed independently. The final loss can be calculated as follows:
C - (C (B, B2) + Larounding B.)) + (B))
ggeneration (T,T):
where / denotes the ground-truth bounding boxes, T and T are the ground-truth captions and generated text predictions from a vision-language model in LLMDet [T] respectively.
4 Experiments
4.1
Implementation Details
For VL-SAM, we choose VL.ernVL.-2.5---- with Intern ViT-300M [6] and InternI.M2.5-78 [5] as the vision-language model. We set the temperature to 0.8 and top-p for nucleus sampling to 0.8 for VnterVV--2.5--0 For the open-set model, we select LLMDet [ as the baseline model because of its SOTA performance. The number N of additional learnable queries is set to 900. For denoising points, both hyper-parameters X and X2 are set to 1.
The whole model of VL-SAM-VE V2 is fine-tuned with GroundingCap- 1M dataset [1] following the training protocol of LLMDet . During training, only the self-attention modules in general and specific query fusion and box heads are fine-tuned, while others are frozen. We fine-tune VL-SAM-V2 for 150k iterations using automatic preed-preciioon with batch size of 16. All training can be done on 8 NVIDIA A800 GPUs within two days.
4.2 Main Results
We mainly evaluate the proposed method on the LVIS dataset, which contains 1203 categories. We adopt the fixed AP (8] as the evaluation metric on frequent, common, and rare classes
Open-set Object Detection. As shown in Table VL-SAM-V2 beats all previous open-set models and achieves the new state-of-the-ar zero-shot open-set object detection results. Specifically, with general and specific query fusion, VL-SAM-VE V2 outperforms the baseline LLMDet by 1.0 AP and 0.8 AP on LVIS minival with Swin-T 30 and Swin-L as backbones, respectively. Notably, for rare objects, VL-SAM-V2 obtains significant performance improvement, from 37.3 AP, to 41.2 AP, with Swin-T and from 45.1 AP, to 47.2 AP, with Swin-L. These results demonstrate our motivation that the prior knowledge from large vision-language models enables the open-set model discover rare objects.
In addition, notice that DeICLIPv3 4] achieves higher AP, and APc. The reason is that DeICLIPY3 collects more balanced data and noun concept corpora. Nevertheless, we believe that the performance of DetCLIPy3 can be further improved by the fusion pipeline of VL-SAM-V2.
Moreover, we evaluate VL-SAM-V2 on other datasets, including COCO 26 and CODA (2). As shown in Table 3, our method achieves the best results on various datasets, especially on the CODA dataset.
Open-ended Object Detection. For open-ended object detection evaluation, we set the predefined object category list to empty and follow GenerateU [25] and VL-SAM (27] to match the generated object categories with the category list in LVIS by CLIP 33 We present the open- open-ended object detection results in Table The results show that VL-SAM-V2 achieves the best open-ended object detection performance on AP and AP,. Specifically, with Swin-TT T as the backbone, VL-SAM-V2 outperforms GenerateU by 2.7 AP and 9.8 AP,. For Swin-L, VL-SAM-V2 also improves AP, by a large margin compared to previous methods, from 22.3 AP, and 23.4 AP, to 30.5 AP,. Moreover, though VI-SAM obtains higher AP, than GenerateU, the overall AP lower than GenerateU. This indicates that the priority of vision- language models favors rare objects but performs unsatisfactorily

---

Table 1: Comparison of zero-shot open-set object detection results LVIS val and minival 13
We report fixed AP [ Grey results denote using additional private data.
LVISminival
LVIS
Method
Backbone
AP
AP,
AP.
AP,
AP
AP,
AP.
AP;
GLIP
24
Swin-T
26.0
20.8
21.4
31.0
17.2
10.1
12.5
25.2
GLIPv2
Swin-T
29.0
CapDet
31
Swin-T
33.8
29.6
32.8
35.5
Grounding-DINO 29
Swin-T
27.4
18.1
23.3
32.7
20.1
10.1
15.3
29.9
OWL-ST 32
CLIP B/16
34.4
38.3
28.6
30.3
Desco-GiL.P
Swin-T
34.6
30.8
30.5
39.0
26.2
19.6
22.0
33.6
DetCLIP
40
Swin-T
35.9
33.2
35.7
36.4
28.4
25.0
27.0
28.4
DetCLIPv2
39
Swin-T
40.4
36.0
41.7
40.4
32.8
31.0
31.7
34.8
HOLO-Worlde
YOLOv8-L
35.4
27.6
34.1
38.0
T-Rex2 17
Swin-T
42.8
37.4
39.7
46.5
34.8
29.0
31.5
41.2
OV-DINO 37
Swin-T
40.1
34.5
39.5
41.5
32,9
29.1
30.4
37.4
LLMDet
Swin-T
44.7
37.3
39.5
50.7
34.9
26.0
30.1
44.3
VL-SAM-V2 (Our)
Swin-T
45.7
41.2
41.1
50.7
35.5
29.3
31.8
44.3
GLIP 24
Swin-L
37.3
28.2
34.3
41.5
26.9
17.1
23.3
36.4
GLIPv2
Swin-H
50.1
Grounding DINO
29
Swin-L
33.9
22.2
30.7
38.8
OWL-ST 32
CLIP
40.9
41.5
35.2
36.2
DetCLIP 40
Swin-L
38.6
36.0
38.3
39.3
28.4
25.0
27.0
31.6
DeICL.PP22 39
Swin-L.
44.7
43.1
46.3
43.7
36.6
33.3
36.2
38.5
DeCCLIPP33 44
Swin-L.
48.8
49.9
49.7
47.8
41.4
41.4
40.5
42.3
LLMDDet 11
Swin-I.
51.1
45.1
46.1
56.6
42.0
31.6
38.8
50.2
VL-SAM-V2 (Our)
Swin-L
51.7
47.2
46.7
56.6
42.5
33.2
39.7
50.2
Table 2: Comparison of zero-shot open-ended object detection results LVTS minival [3. We report,fied AP 8. VL-SAM utilizes ViT-H [T0 of SAM as the image encoder for segmentation.
Method
Image Encoder
Vision-Language Model
AP
AP APrare
GenerateU 25
Swin-T Swin-T
FlanT5- Intern VL- (8B)
26.8
20.0
VL-SAM- V2 (Our)
29.5
29.8
GenerateU 23
Swin-L
F5anT5- base
27.9
22.3
VL-SAM 27
ViT-H
CogVLM (17B)
25.3
23.4
VL-SAM- (Our)
Swin-L
InternVL-2.5 (8B)
31.8
30.5
on frequent objects. contrast, VL-SAM-V2 obtains both the best AP and AP,, demonstrating that the fusion with the general proposals from open-set models and specific proposals from vision- language models helps the models to retain the high performance of open-set models on frequent objects.
4.3 Ablation Study
In this section, we conduct ablation experiments to analyze the main components and model general- ization of VL-SAM-V2, We use Swin-T I as the backbone and report the results on LVIS minival.
Main Components. To evaluate the effectiveness of each component, we successively add compo- nents the LLMDet baseline. As shown in Table we observe that each component consistently improves performance. Specifically, adding general and specific query fusion brings improvements of 0.5 AP, 1.8 AP,, and 0.8 AP,. respectively. Then, we adopt ranked learnable queries to replace the single learnable query for each point prompt, obtaining a 1.1 AP, improvement. Moreover, training with denoising points can further improve AP, by 1.0. Meanwhile, the overall training cost reduced by 50% due to the removal of point generation VL-SAM.

---

Table 3: Comparison of open-set object detection results on COCO and CODA. We report mAP for each dataset.
Method
COCO
CODA
GroundingDINO
48.4
12.6
YOLO-Worke
45.1
16.1
LLMDet
55.6
38.5
VL-SAM-V2
56.0
42.3
Table 4: Ablations on main components. 'GS Fusion' denotes the general and specific query fusion module. Each component improves the detection performance consistently.
GS Fusion Ranked Learnable Queries
Denoising Points
AP
AP,
AP.
44.7
37.3
39.5
45.2
39.1
40.3
45.4
40.2
40.6
45.7
41.2
41.1
Table Ablations on fusion methods. 'GS Fusion' denotes the general and specific query fusion module.
Fusion methods
AP
AP,
Naive Concatenate
44.9
38.5
Late Fusion
45.1
40.2
GS Fusion
45.7
41.2
Overall, the results demonstrate the effectiveness of each component proposed in VL-SAM-V2.
Fusion Methods. In addition to the proposed general and specific query fusion, try other fusion methods, i..., Naive Concatenate and Late Fusion. Naive Concatenate simply concatenates two query features into one feature with linear layer. Late Fusion directly ensembles the detection results from two queries. As shown in Table 5 we can observe that GS Fusion achieves the best fusion performance, outperforming naive concatenate by 2.7 AP,
To demonstrate the model generalization of the VL-SAM-V2, adapt various popular open-set models and vision-language models, including Grounding 29 LLMDet (II), LLaVA (28), VLM [38), and Intern [5]
Model Generalization.
As shown Table 6 VL-SAM-V2 consistently improves the performance of different open-set models with different vision lision-language models, ranging from 2.3 to 4.9 AP,. Moreover, the empirical results show that stronger vision- languuge models can obtain better detection performance, aligning with the finding in VL-SAM (27). This indicates that VL-SAM-V2 can benefit from more powerful open-set models and vision- lingunl models.
4.4 Combined with SAM.
Following Grounded-SAN [35, we combine the proposed VL-SAM-V2 with the segment-anything model (SAM) 18 to achieve open- sorld instance segmentation.
As shown in Table we compare the proposed method combining SAM with VL-SAM on the open-ended instance segmentation task. We can find that our method shows favorable performance improvement. In addition, we visualize the open-ended instance segmentation results on the corner case object detection dataset, CODA, to demonstrate the effectiveness of the proposed method in real-world scenarios. As depicted Figure[4] our method can provide dense segmentation masks and category annotations for input images and discover various novel object categories outside of the existing autonomous driving datasets [36] [. including sacks and cranes. Moreover, since VL-SAM-V2 uses point prompts as inputs, users can provide point prompts by themselves to achieve

---

Table 6: Ablation of model generalization. VL-SAM-V2 can adopt various open-set models and vasion-language models for point prompting. f denotes results are reimplementated by MMDetec- tion (2.
Method
Open-set Models
Vision-Language Model
AP
APrare
GroundingDINOf LLMDet
41.4
34.2
44.7
37.3
LLaVA (7B)
42.1
37.3
GroundingDINO
CogVLM (17B)
42.7
38.5
InternVI-22.5 (8B)
43.0
39.1
VI-SAAM-2 (Our)
VL (7B)
45.2
39.6
LLMDet
Cog (17B)
45.5
40.6
InterVL-2..0 (8B)
45.7
41.2
Table 7: Comparison of zero-shot open-ended instance segmentation results on LVIS mini- val [3].
Method
Vision-Language Model
mask AP
mask AP pare
VL-SAM
CogVLM (17B)
22.7 23.9
VL-SAM-V2 V2 (Our)
InterVVL.2.. (8B)
28.7
27.7
Figure Visualization results VL-SAM-V2 combining with SAM CODA (22). We show input images and detection and segmentation prediction results in the open-ended mode. VL-SAM-V2 can discover various uncommon objects. Best viewed by zooming in.
prompt-based detection and segmentation, demonstrating the diverse applications the proposed method in real-world scenarios.

---

Limitations
Since we adopt the VL-SAM framework to utilize vision-language models to generate point prompts, VL-SAM-V2 inherits the defects of vision-language models, including the hallucination problem, incorrect responses, and low inference speed. However, these defects will be gradually overcome with the development of vision-language models. The proposed VL-SAM-V2 framework can benefit from these new models. Moreover, VL-SAM-V2 needs additional SAM to obtaín instance segmentation results. In the future, we will integrate segmentation models into our framework to compose an end-to-end open-world perception model.
Conclusion
In this work, introduce VL-SAM-V2, novel open- speldworld object detection framework that perform in the open-set or open-ended mode. The core of VL-SAM-V2 is the general and specific query fusion that integrates information from both open-set and open-ended perception paradigms. To further enhance the model's ability to discover unseen objects, we present ranked learnable queries promote more diverse and representative query representations. Additionally, propose the denoising point training strategy to stabilize and facilitate the training process. Experimental results on the LVIS demonstrate that VL-SAM-V2 outperforms previous state-of-the-art methods in both open-set and open-ended settings, particularly excelling in detecting rare objects. Moreover, VL-SAM-VV V2 exhibits good model generalization that can incorporate various open-set models and vision-language models. By combining with SAM, VL-SAM-VE V2 shows promise for new generation of automatic labeling and open-world perception systems.
Broader Impacts Statement. This paper investigates the use of current open-set models and vision- language models for open- world object detection. We do see potential privacy-related issues. This study may inspire future research on automatic labeling systems and open-world perception models.
Acknowledgments. This work was supported National Key R&D Program of China (Grant No. 2022ZDD0100355).
10

---

References
[1] Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset autonomous driving. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[2] Kai Chen, Jiaqi Wang. Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun. Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: I 1906,07155, 2019.
[3] Ting Chen, Simon Korblithh Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning (ICML), 2020.
[4] Ting Chen, Saurabh Saxena, Lala Li, David Fleet, and Geoffrey Hinton. Pix2seq: language modeling framework for object detection, arXiv preprint arXiv:2109. 10852, 2021.
[5] Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye, Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.
[6] Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su. Guo Chen, Sen Xing. Muyan Zhong, Qinglong Zhang. Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic vinual-linguistc tasks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024,
[7] Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real- time open-vocabulary object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[8] Achal Dave, Piotr Dollár, Deva Ramanan, Alexander Kirillov, and Ross Girshick. Evaluating large- vocabulary object detectors: The devil is in the details. arXiv preprint urXiv:2102.01006 2021.
[9] Akshay Raj Dhamija, Manuel Günther, and Terrance Boult. Reducing network agnostophobia. Neural Information Processing Systems (NeurIPS), 2018.
[10] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition seale. In International Conference on Learning Representations (ICLR), 2020.
[11] Shenghao Fu, Qize Yang. Qijie Mo, Junkai Yan, Xihan Wei, Jingke Meng. Xiaohua Xie, and Wei-Shi Zheng. LImdet: learning strong open-vocabulary object detectors under the supervision of large language models. arXiv preprint 2r0iv:2501. 18954, 2025.
[12] Chuanxing Geng, Sheng-jun Huang, and Songcan Chen. Recent advances in open set recognition: A survey. IEEE Transactions on Pattern Recognition and Machine Intelligence (PAMI), 2020.
[13] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[14] Akshita Gupta, Sanath Narayan, KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Ow- detr: Open-worts detection transformer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[15] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In IEEE Conference on Computer Vision and Puttern Recugnition (CVPR), 2020.
[16] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-enn. In IEEE International Conference on Computer Vision (ICCV). 2017.
[17] Qing Jiang, Feng Li, Zhaoyang Zeng, Tianhe Ren, Shilong Liu, and Lei Zhang. T-rex2: Towards generic object detection via visual prompt synergy. Earupean Conference on Computer Vision (ECCV), 2024.
[18] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Yen Lo, et al. Segment anything. In IEEE International Conference Computer Vision (ICCV), 2023.

---

[19] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[20] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Ii, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0333 2024.
[21] Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang. Dn-detr; Accelerate detr training by introducing query denoising. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[22] Kaican Li. Kai Chen, Haoyu Wang, Lanqing Hong, Chaoqiang Ye. Jianhua Han, Yukuai Chen, Wei Zhang. Chunjing Xu, Yan Yeung, et al. Coda: A real-world road corner case datasct for object detection in autonomous driving. In European Conference on Computer Vision (ECCV), 2022.
[23] Liunian Li, Zi-Yi Dou, Nanyun Peng, and Kai-Wei Chang. Desco: Learning object recognition with rich language descriptions. Neural Information Processing Systems (NeurIPS), 2023.
[24] Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, ct al. Grounded language-image pre-training. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[25] Chuang Lin, Yi Jiang. Lizhen Qu, Zehuan Yuan, and Jianfei Cai. Generative region-languag pretraining for open-ended object detection, In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[26] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.
[27] Zhiwei Lin, Yongtao Wang, and Zhi Tang. Training-free open-ended object detcction and segmentation via attention as prompts. Neural Information Processing Systems (NeurIPS). 2024.
[28] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Neural Information Processing Systems (NeurlPS), 2023.
[29] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang. Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision (ECCV), 2024.
[30] 7e Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In IEFF International Conference on Computer Vision (ICCV). 2021.
[31] Yanxin Long, Youpeng Wen, Jianhua Han, Hang Xu, Pengzhen Ren, Wei Zhang, Shen Zhao, and Xiaodan Liang- Capdet: Unifying dense captioning and open-world detection pretraining. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[32] Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. Neural Information Processing Systems (NeurIPS), 2023.
[33] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
[34] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real- time object detection with region proposal networks. Neural Information Processing Systems (NeurIPS), 2015.
[35] Tianhe Ren, Shilong Liu, Ailing Zeng. Jing Lin, Kunchang Li. He Cao. Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, al. Grounded sam: Assembling open-world models for diverse visual tasks, arXiv preprint arXiv:2401. 14159, 2024.
[36] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, Vijay Vasudevan, Wei Han, Jiquan Ngiam, Hang Zhao, Aleksei Timofeev, Scott Ettinger, Maxim Krivokon, Amy Gao, Aditya Joshi, Yu Zhang, Jonathon Shlens, Zhifeng Chen, and Dragomir Anguelov. Scalability in perception for autonomous driving: Waymo open dataset. In IEEE Conference Computer Vision and Pottern Recognition (CVPR), 2020.
12

---

[37] Hao Wang, Pengzhen Ren, Zequn Jie, Xiao Dong, Chengjian Feng, Yinlong Qian, Lin Ma. Dongmei Jiang. Yaowei Wang, Xiangyuan Lan, al. Ov-dino: Unified open-vocabulary detection with language-aware selective fusion. arXiv preprint urXiv:2407.091 077444 2024.
[38] Weihan Wang. Qingsong Lv, Wenmeng Yu, Wenyi Hong. Ji Qi. Yan Wang. Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, Cogylm: Visual expert pretrained language models. Neural Information Processing Systems (NeurIPS), 2024.
[39] Lewei Yao. Jianhua Han, Xiaodan Liang, Xu, Wei Zhang, Zhenguo Li, and Hang Xu. Detclipv2: Scalable open vocabuuayy object detection pre pre-training via word-region alignment. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2023.
[40] Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detelip: Dictionary-enriched visual-concept paralleled pre-training for open-world detection. Neural Information Processing Systems (NeurIPS), 2022.
[41] Lewei Yao, Renjie Pi, Jianhua Han, Xiaodan Liang, Hang Xu, Wei Zhang, Zhenguo Li, and Dan Xu. Detelipv3: Towards versatile generative open-vocabulary object detection. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2024.
[42] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang. Hang Su, Jun Zhu, Lionel M. Ni, and Heung- Yeung Shum. DINO: DETR with improved denoising anchor boxes for end-to and-to-end object detection. In International Conference on Learning Representations (ICLR), 2023.
[43] Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Leizhang, Chunyuan Li, et al. Llava European Conference on Computer Vision (ECCV), 2024.
Llava-groumde grounding: Grounded visual chat with large multimodal models. In
[44] Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang. Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. Neural Information Processing Systems (NeurIPS), 2022.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant
information is Not Available.
Please provide a short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked "[Yes] is generally preferable to "[No] ". is perfectly acceptable to answer "[No] provided a proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset used"). In general, answering "[No] oo "[NA] is not phrased in binary way. acknowledge that the answer write a justification elaborate.
the checklist as of the factors in their evaluation. While
grounds for rejection. While the questions often more nuanced, so please just use your best judgment evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
supporting
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify questions and only use the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We claim the main contribution of this paper in both the Abstract and Introduction
sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can he expected to generalize other setrings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of this work in Section S
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged create separate "Limitations" section in their paper.
14

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification. asympiotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e-g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicaa,,,, the authors should discuss possible limitations of their approach to address problems of privacy and fairness,
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory assumptions and proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the implementation details in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
15

---

While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. . a large language model), then there should either be a way to access this model for reproducing the results a way to reproduce the model (e.g.. with open-source datasct instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of booeddsouuce models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other rescarchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does paper provide open access to the data and code. with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We not provide new datasets and will release partial code after the paper accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (httpa://nips. httpaa//niip.ccppubllee cc/public/ Sttpe//oooSuuuissiiiiiees
for more details.
While we encourage the release of code and data, we understand that this might not be possihle, so "No" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines hattpa://nipe http:///iips.ccppp/epeee cc/public/ htteee//oodSttmiisssoooooooi for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparamcters, how they were chosen, type of optimizer. etc.) necessary to understand results?
Answer: [Yes]
Justification: We provide training details and hyperparameters in Section 4.1
Guidelincs:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars not reported because it would be too computationally expensive.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The authors should answer "Yes" if the results are accompanied by enoor bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims
the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given
experimental conditions).
method for caleulating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigaa error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be carcful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates). error bars reported tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8, Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: provide information for computer resources in Section 4.1 Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code Ethics http::://oorrr.....ppptttttttttt cc/ppubllc/hbsseeeueeel...ee
Answer: [Yes]
Justification:
The research in the paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the discussion of broader impacts in Section O
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g..
disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arisc when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data models that have high risk for misuse (e.g. pretrained language models, image generators, scraped datascts)?
Question:
Answer: [NA]
Justification:
The models in this paper pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All owners of models, code, and data we used are properly cited. We compliance all licenses of models, code, and data.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppappppteeecco thcode. com/ datasets has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release assets.
18

---

Researchers should communicate the details of the dataset/code/mooel as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL. or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should included the main paper.
According to the NeurlPS Code Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing rescarch with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this the paper.
recognize that procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
16. Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, non-standard methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
component of the
Answer: [Yes]
Justification: We describe the usage of LLMs in Section
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components.
Please refer to our L.M policy (https: :/ http://eeurrpppppe http://eeurrrpp..oooooeeeeeeeeee www.oo/e//eneeeeeeccc 201.c/eeeeee222/211//// LLM) for what should or should not be described.
19


Reviews:
[{"review_id": "QwX47Kglew", "reviewer": "Reviewer_1JWr", "summary": "This paper proposes VL-SAM-V2 for open-world object detection that aims to combine the complementary strengths of open-set and open-ended paradigms.\nIt introduces a General and Specific Query Fusion module that enables interaction between specific queries from an open-set detector and general queries derived from VL-SAM. \nIt introduces ranked learnable queries, which associate object point prompts with learnable object queries based on importance scores.\nIn addition, a denoising points training strategy is devised to inject noise into sampled points to stabilize training and reduce reliance on expensive VL-SAM inference.", "questions": "See weakness", "limitations": "See weakness", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nEvaluation is extensive.\n\nComponent-level ablation is provided.\n\nWeaknesses:\n\nWhile the paper proposes denoising points to avoid reliance on VL-SAM during training, it never quantifies the cost (runtime, memory, VLM overhead) of using VL-SAM at inference time.\n\nThere is little insight into when VL-SAM-V2 fails (e.g., misdetections, hallucinations from VLMs).\n\nThe distinction between “point prompts,” “noisy points,” and “ranked queries” can become confusing.\n\nWhile the architecture is explained, it's not always clear why the joint attention mechanism improves over naive concatenation or late fusion. \n\nThe entire evaluation is centered on LVIS, performance on other open-world settings (e.g., COCO-OOD or autonomous driving scenarios) is not provided. \n\nThe integration with SAM is demonstrated, but not compared with other segmentation alternatives. It's unclear if SAM is critical or replaceable.\n\nAlthough the fusion of query types is novel, the individual components (like denoising point training and query ranking) are adaptations of existing techniques (e.g., DETR denoising, anchor assignment). \n\nMuch of the performance gain may stem from powerful pretrained VLMs (e.g., InternVL-2.5), how much of the improvement is due to the method rather than the foundation model?", "was_revised": true, "final_justification": "The authors have addressed most of the concerns raised.", "rebuttal": "We thank the reviewers for their constructive comments. We address the reviewers' concerns in the rebuttal text below.\n### **Q1: Inference cost**\nWe test the inference time and GPU memory consumption of VL-SAM-V2 on a single NVIDIA A800 GPU. We use Swin-T as the backbone and InternVL-2.5-8B as the VLM. We can find that the main inference time and GPU memory are spent on the VL-SAM to generate general queries.\n\n| module                            | inference time | GPU memory |\n| --------------------------------- | -------------- | ---------- |\n| general query generation (VL-SAM) | 1.2 s          | ~72 G      |\n| specific query generation         | 17 ms          | ~10 G      |\n| Fusion & Head                     | 5 ms           | ~2 G       |\n| Overall                           | 1.2 s + 22 ms  | ——         |\n\n### **Q2: Failure case**\nAs discussed in the Limitations Section, since we utilize vision-language models to generate point prompts, VL-SAM-V2 may fail to predict correct results when VLMs give incorrect responses (including hallucinations). Specifically, when hallucinations happen, VLMs can give wrong object names or point locations, leading to incorrect predictions. In addition, VLMs may miss objects in the image. For instance, as shown in the last picture of Figure 4 in the paper, VLMs fail to give the \"Tree\" class. Subsequently, our method can not give a prediction of “Tree”.\n\n### **Q3: Confusion between “point prompts”, “noisy points”, and “ranked queries”**\n\nSorry for the confusion caused by the unclear names. \"point prompts\" denotes the points priority generated from VL-SAM. \"noisy points\" is the sampled points within ground-truth bboxes by Eq.1 in the paper. \"ranked queries\" represents learnable queries, which will be matched with points (including \"point prompts\" and \"noisy points\") by ranking before sent into GS Fusion.\n\nWe will give a more precise definition in the paper.\n\n### **Q4: Joint attention mechanism improves over naive concatenation or late fusion**\nWe replace our GS Fusion with other fusion methods, including naive concatenate and late fusion. The results of different fusion methods for open-set detection are shown in the table. We can observe that GS Fusion achieves the best fusion performance, outperforming naive concatenate by 2.7 AP (rare).\n\n| Fusion methods    | AP   | AP (rare) |\n| ----------------- | ---- | --------- |\n| Naive concatenate | 44.9 | 38.5      |\n| Late fusion       | 45.1 | 40.2      |\n| GS Fusion (Ours)  | 45.7 | 41.2      |\n\n### **Q5: COCO or autonomous driving scenarios evaluation**\nWe provide the open-set evaluation results on LVIS, COCO, and CODA[1]. The results show that our method achieves the best results on various datasets, demonstrating the effectiveness of VL-SAM-V2.\n\n| Method        | LVIS | COCO | CODA |\n| ------------- | ---- | ---- | ---- |\n| GroundingDINO | 27.4 | 48.4 | 12.6 |\n| YOLO-World    | 35.4 | 45.1 | 16.1 |\n| LLMDet        | 44.7 | 55.6 | 38.5 |\n| VL-SAM-V2     | 45.7 | 56.0 | 42.3 |\n\n[1] Li K, Chen K, Wang H, et al. Coda: A real-world road corner case dataset for object detection in autonomous driving. European conference on computer vision.\n\n### **Q6: Compared with other segmentation alternatives**\n\nOur method can be technically combined with any point/box prompt methods. As shown in the table, replacing SAM with other segmentation methods, including Efficient SAM[1] and GLEE[2], can achieve favorable segmentation performance.\n\n| Segmentation model | LVIS mask AP |\n| ------------------ | ------------ |\n| SAM                | 28.7         |\n| Efficient SAM      | 25.5         |\n| GLEE               | 28.4         |\n\n[1]Zhang Z, Cai H, Han S. Efficientvit-sam: Accelerated segment anything model without performance loss. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n[2]Wu J, Jiang Y, Liu Q, et al. General object foundation model for images and videos at scale. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\n### **Q7: Novelty about denoising point training and query ranking**\n\nFor denoising point training, there are two key differences compared to DETR denoising training. First, our training strategy directly predicts boxes from noisy points, while DETR predicts box offsets from noisy boxes. Second, the motivation of our training strategy is to reduce reliance on expensive VL-SAM inference. In contrast, DETR denoising training is often used to solve the instability of bipartite matching.\n\nFor query ranking, we are actually motivated by the anchor assignment in DETR. In DETR, the content queries are selected from features. Thus, the queries are naturally combined with the score. In contrast, our VL-SAM-V2 only has points. To match points with the queries, we follow DETR to predict the score of each point.\n\n### **Q8: How much of the improvement is due to the method rather than the foundation model**\n\nWe directly evaluate the open-ended object detection performance of the foundation models (including InternVL-2.5-8B, QwenVL-2.5-7B, and LLaVA-OneVision-7B). We follow the official spatial understanding inference code of QwenVL-2.5-7B to obtain the detection results. As shown in the table, VL-SAM-V2 outperforms InternVL-2.5 by a large margin. These results demonstrate that simply adopting the foundation model can not obtain good detection results. In contrast, adopting general and specific fusion in VL-SAM-V2 can boost the performance, showing the effectiveness of the proposed method.\n\n| method            | AP   |\n| ----------------- | ---- |\n| InternVL-2.5-8B   | 6.21 |\n| QwenVL-2.5-7B     | 6.53 |\n| LLaVA-OneVison-7B | 4.22 |\n| VL-SAM-V2 (Ours)  | 29.5 |"}, {"review_id": "MS8p4IfQqz", "reviewer": "Reviewer_PGzh", "summary": "This work, VL-SAM-V2 is an open-world object detection framework that unifies the strengths of open-set (category-guided) and open-ended (category-free) detection. It introduces a general and specific query fusion module to combine proposals from both open-set detectors and vlms. To train efficiently, it uses a denoising point strategy, simulating noisy object prompts from ground-truth boxes instead of relying on expensive VL models like InternVL at every step. It also proposes ranked learnable queries that assign stronger query vectors to higher-confidence points. VL-SAM-V2 achieves improved results on the LVIS benchmark in both open-set and open-ended modes. The framework is modular and adaptable, supporting various VL backbones and detectors, and also integrates with SAM for instance segmentation.", "questions": "How is the query fusion different from GLIP-style attention? What is novel about applying DINO's denoising strategy to points? How does VL-SAM-V2 compare to recent VLMs like Qwen2-VL or LLaVA-OneVision in open-ended detection? Can the authors provide results on COCO to assess generalization? A failure case analysis and more detailed ablations (See weakness for details.)", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "## Strength\n\nMain strength lies in its unified design that combines open-set and open-ended detection via a lightweight query fusion module, enabling strong performance across both frequent and rare object classes. Its use of ranked learnable queries and denoising point training ensures efficient, robust learning without relying on expensive VL models during training, while retaining modularity to plug in stronger vision-language and detection backbones.\n\n## Weakness\n\n- How is your query fusion different from what models like GLIP/GLIPv2 already do? Isn't it just attention between two sets of queries, similar to their cross-modal fusion?\n- Denoising point strategy follows the denoising anchor box idea from DINO, what is the actual novelty or benefit introduced here?\n- Since VL-SAM-V2 aims to handle open-ended detection, it would be helpful to understand how it compares to more recent vision-language models like Qwen2/LLaVA-OneVision, which also support grounding. Could the authors provide these baselines and comment on how their approach improves over such models?\n- While LVIS is a strong benchmark, open-world models are also expected to generalize across datasets. Could the authors provide results on COCO settings to assess cross-domain robustness?\n- The qualitative results mostly highlight successful detections, but to better understand the model's limitations, could the authors provide an analysis of failure cases?\n- Could the authors offer more detailed analysis? For example, how does performance vary with the number of ranked queries, what is the sensitivity to the $\\lambda$1 and $\\lambda$2 parameters used in denoising point sampling, and how do the general and specific query branches perform individually?", "was_revised": true, "final_justification": "The authors have provided helpful clarifications and additional analysis that address my main concerns. The revisions strengthen the overall contribution, and I have updated my score.", "rebuttal": "We thank the reviewers for their constructive comments. We address the reviewers' concerns in the rebuttal text below.\n\n### **Q1: How is your query fusion different from GLIP/GLIPv2**\n\nThe key difference is the similarity generation during the attention mechanism. GLIP/GLIPv2 adopts cross-modality similarity while VL-SAM-V2 uses full similarity. Specifically, given two different queries, q1 and q2, the similarity matrix s in GLIP/GLIPv2 is calculated with $$s=q_1\\times q_2^T.$$ Then, q1 and q2 are updated by\n$$\nq_1=softmax(s)\\times W_2q_2;~~~~~~~q_2=softmax(s^T)\\times W_1q_1,\n$$\nwhere W1 and W2 are learnable weights.\nIn contrast, we first concatenate q1 and q2:\n$$\nq=concat([q_1,q_2])\n$$\nThen, we calculate self-attention for q and split the updated q into q1 and q2\n\n### **Q2: Novelty or benefit of denoising point strategy**\n\nThere are two key differences between the denoising point strategy and DINO denoising training. First, our training strategy directly predicts boxes from noisy points, while DINO predicts box offsets from noisy boxes. Second, the motivation of our training strategy is to reduce reliance on expensive VL-SAM inference. In contrast, DINO denoising training is often used to solve the instability of bipartite matching.\n\n### **Q3: Open-ended detection performance of recent vision-language models like Qwen2/LLaVA-OneVision**\nWe report the open-ended detection performance of recent VLMs, including InternVL-2.5-8B, QwenVL-2.5-7B, and LLaVA-OneVision-7B, on LVIS. We follow the official spatial understanding inference code of QwenVL-2.5-7B to obtain the detection results. The results show that recent vision-language models perform poor in open-ended multi-object detection.\n\n| method            | AP   |\n| ----------------- | ---- |\n| InternVL-2.5-8B   | 6.21 |\n| QwenVL-2.5-7B     | 6.53 |\n| LLaVA-OneVision-7B | 4.22 |\n| VL-SAM-V2 (Ours)  | 29.5 |\n\n### **Q4: Results on COCO settings and other benchmarks**\nWe provide the open-set evaluation results on LVIS, COCO, and CODA[1]. The results show that our method achieves the best results on various datasets, demonstrating the effectiveness of VL-SAM-V2.\n\n| Method        | LVIS | COCO | CODA |\n| ------------- | ---- | ---- | ---- |\n| GroundingDINO | 27.4 | 48.4 | 12.6 |\n| YOLO-World    | 35.4 | 45.1 | 16.1 |\n| LLMDet        | 44.7 | 55.6 | 38.5 |\n| VL-SAM-V2     | 45.7 | 56.0 | 42.3 |\n\n[1] Li K, Chen K, Wang H, et al. Coda: A real-world road corner case dataset for object detection in autonomous driving. European conference on computer vision.\n\n### **Q5: Analysis of failure cases**\nAs discussed in the Limitations Section, since we utilize vision-language models to generate point prompts, VL-SAM-V2 may fail to predict correct results when VLMs give incorrect responses (including hallucinations). Specifically, when hallucinations happen, VLMs can give wrong object names or point locations, leading to incorrect predictions. In addition, VLMs may miss objects in the image. For instance, as shown in the last picture of Figure 4 in the paper, VLMs fail to give the \"Tree\" class. Subsequently, our method can not give a prediction of “Tree”.\n\n### **Q6: More detailed ablation analysis**\nWe first provide the ablation studies of the number of queries. We can find that the performance of VL-SAM-V2 improves when the number of queries increases. However, more queries bring more inference time. Thus, we choose 900 queries as the default setting, as it achieves a good efficiency-accuracy trade-off.\n\n| number of queries | AP   | inference time |\n| ----------------- | ---- | -------------- |\n| 900               | 45.7 | 1 x    |\n| 600               | 45.2 | 0.8 x  |\n| 1200              | 45.8 | 1.2 x  |\n\n\n\nThen, we provide the ablation studies of lambda in the following table. The results show that VL-SAM-V2 is robust to lambda_2. For lambda_1, VL-SAM-V2 performs well when lambda_1 is smaller than 1.0. Increasing lambda_1 above 1.0 may introduce false positive points during denoising training, hurting overall performance.\n\n| lambda_1 | lambda_2 | AP   |\n| -------- | -------- | ---- |\n| 1.0      | 1.0      | 45.7 |\n| 0.5      | 1.0      | 45.3 |\n| 1.5      | 1.0      | 44.1 |\n| 1.0      | 0.5      | 45.5 |\n| 1.0      | 1.5      | 45.4 |"}, {"review_id": "hyaHxrQuTM", "reviewer": "Reviewer_JcR6", "summary": "This paper presents VL-SAM-V2, an open-world object detection framework that combines open-set and open-ended detection paradigms through a general and specific query fusion mechanism. The method uses VL-SAM to generate general queries for discovering novel objects and LLMDet for specific queries targeting predefined categories. Key technical contributions include a query fusion module, ranked learnable queries, and a denoising point training strategy. Experiments on LVIS demonstrate improvements over existing methods in both open-set and open-ended settings, with particularly notable gains on rare objects.", "questions": "Are there cases where the fusion actually hurts performance compared to individual methods?\nCan you provide results on additional datasets to demonstrate broader applicability? LVIS has specific characteristics (long-tail distribution) that may favor your approach.\nAblation on fusion mechanism: Have you experimented with alternative fusion strategies beyond self-attention? How sensitive is performance to the specific fusion architecture?", "limitations": "The authors adequately discuss limitations in Section 5, acknowledging inheritance of vision-language model defects including hallucination and inference speed issues. However, the limitations section could be expanded to discuss:\nDependency on quality of underlying vision-language models\nPotential brittleness of the multi-component pipeline\nLimited analysis of computational trade-offs\nEvaluation scope constraints", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\nWell-motivated problem: The paper addresses a practical limitation where open-set models require human-provided category lists while open-ended models suffer from poor performance on frequent objects. The proposed fusion approach is intuitive and well-motivated.\nComprehensive experimental evaluation: The paper includes thorough ablation studies demonstrating the contribution of each component, generalization experiments across different open-set models and vision-language models, and analysis of computational requirements.\nTechnical soundness: The query fusion mechanism using self-attention is reasonable, and the ranked learnable queries provide a principled way to match point prompts with learned representations.\nWeaknesses:\nLimited technical novelty: The core contribution is essentially combining two existing methods (VL-SAM and LLMDet) with a relatively straightforward fusion module. While effective, the technical innovation is incremental.\nI still hope the authors can further elaborate on the differences between VL-SAMv2 and simply combining the existing VL-SAM and LLMDet methods.\nComplex pipeline: The method involves multiple components (VL-SAM, open-set model, fusion module, denoising strategy) which makes it complex to implement and potentially brittle. The paper doesn't adequately discuss failure modes or sensitivity analysis.\nLimited evaluation scope: Experiments are conducted only on LVIS. Evaluation on other datasets (e.g., COCO, Objects365) would strengthen the claims about generalizability.\nComputational overhead not thoroughly analyzed: While the paper mentions training costs are reduced by 50% due to removing VL-SAM inference during training, the inference-time computational overhead compared to individual methods is not clearly quantified.", "was_revised": true, "final_justification": "The response partially addressed my concerns, including some additional experiments. However, the innovation appears to be a simple attention-based fusion, which is an incremental innovation. For this reason, I maintain my rating for the current version.", "rebuttal": "We thank the reviewers for their constructive comments. We address the reviewers' concerns in the rebuttal text below.\n\n### **Q1: Differences between VL-SAMv2 and simply combining the existing VL-SAM and LLMDet methods**\n\nIn VL-SAM-V2, we propose a general and specific query fusion module to fuse two distinct types of queries with the attention mechanism in a lightweight manner. To demonstrate the effectiveness of the proposed fusion module, we report simple ensemble results of VL-SAM and LLMDet in the following table. The results show that our fusion method significantly outperform simple ensemble method.\n\n| Fusion methods   | AP   | AP (rare) |\n| ---------------- | ---- | --------- |\n| ensemble         | 45.1 | 40.2      |\n| GS Fusion (Ours) | 45.7 | 41.2      |\n\n### **Q2: Failure modes**\nAs discussed in the Limitations Section, since we utilize vision-language models to generate point prompts, VL-SAM-V2 may fail to predict correct results when VLMs give incorrect responses (including hallucinations). Specifically, when hallucinations happen, VLMs can give wrong object names or point locations, leading to incorrect predictions. In addition, VLMs may miss objects in the image. For instance, as shown in the last picture of Figure 4 in the paper, VLMs fail to give the \"Tree\" class. Subsequently, our method can not give a prediction of “Tree”.\n\n### **Q3: Evaluation on other datasets**\nWe provide the open-set evaluation results on LVIS, COCO and CODA[1]. The results show that our method achieves the best results on various datasets, demonstrating the effectiveness of VL-SAM-V2\n\n| Method        | LVIS | COCO | CODA |\n| ------------- | ---- | ---- | ---- |\n| GroundingDINO | 27.4 | 48.4 | 12.6 |\n| YOLO-World    | 35.4 | 45.1 | 16.1 |\n| LLMDet        | 44.7 | 55.6 | 38.5 |\n| VL-SAM-V2     | 45.7 | 56.0 | 42.3 |\n\n[1] Li K, Chen K, Wang H, et al. Coda: A real-world road corner case dataset for object detection in autonomous driving. European conference on computer vision.\n\n### **Q4: Inference-time computational overhead**\nWe test the inference time and GPU memory consumption of VL-SAM-V2 on a single NVIDIA A800 GPU. We use Swin-T as the backbone and InternVL-2.5-8B as the VLM. We can find that the main inference time and GPU memory is spent on the VL-SAM to generate general queries.\n\n| module                            | inference time | GPU memory |\n| --------------------------------- | -------------- | ---------- |\n| general query generation (VL-SAM) | 1.2 s          | ~72 G      |\n| specific query generation         | 17 ms          | ~10 G      |\n| Fusion & Head                     | 5 ms           | ~2 G       |\n| Overall                           | 1.2 s + 22 ms  | ——         |\n\n### **Q5: Ablation on fusion mechanism**\nWe replace our GS Fusion with naive concatenate and report the results in the following table. We can observe that GS Fusion achieves better fusion performance, outperforming naive concatenate by 2.7 AP (rare). The results demonstrate that the fusion architecture is important in our framework.\n\n| Fusion methods    | AP   | AP (rare) |\n| ----------------- | ---- | --------- |\n| Naive concatenate | 44.9 | 38.5      |\n| GS Fusion (Ours)  | 45.7 | 41.2      |"}, {"review_id": "TogC8x9C13", "reviewer": "Reviewer_6bAf", "summary": "The paper introduces VL-SAM-V2, an open-world object-detection framework that merges the strengths of open-set detectors and open-ended detectors. The system generates general queries from VL-SAM  to cover “everything,” and specific queries from an open-vocabulary baseline to refine frequent classes; a lightweight General & Specific Query Fusion block lets the two query streams co-attend in each decoder layer, while Ranked Learnable Queries and a Denoising Points strategy improve diversity and training stability.", "questions": "Questions\n1. Report end-to-end speed: provide FPS,  memory, and the percentage of latency spent on each subsystem.  \n2. Discuss prior fusion work: how does GS Fusion differ from earlier multi-query DETR or prompt-fusion approaches?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. Solid union of open-set and open-ended cues that demonstrably boosts rare-class recall without hurting frequent classes.\n2. Comprehensive experiments.\n3. Ranked-query and denoising tricks are simple yet effective.\n\nWeaknesses\n1. Figure 2 does not state where the text and image features entering GS Fusion originate.\n2. No end-to-end runtime, memory, or latency numbers.\n3. The paper claims novelty for “general vs specific” fusion but omits discussion of prior multi-source-query detectors.", "was_revised": true, "final_justification": "The author response has addressed most of my concerns; therefore, I will maintain my original rating.", "rebuttal": "We thank the reviewers for their constructive comments. We address the reviewers' concerns in the rebuttal text below.\n\n### **Q1: Figure 2 does not state where the text and image features entering GS Fusion originate**\n\nWe process the text and image features with the open-set model. Specifically, text features are extracted by the BERT encoder, and image features are obtained by a Swin backbone. \n\nWe will add the details in the paper.\n\n### **Q2: Inference cost**\nWe test the inference time and GPU memory consumption of VL-SAM-V2 on a single NVIDIA A800 GPU. We use Swin-T as the backbone and InternVL-2.5-8B as the VLM. We can find that the main inference time and GPU memory are spent on the VL-SAM to generate general queries.\n\n| module                            | inference time | GPU memory |\n| --------------------------------- | -------------- | ---------- |\n| general query generation (VL-SAM) | 1.2 s          | ~72 G      |\n| specific query generation         | 17 ms          | ~10 G      |\n| Fusion & Head                     | 5 ms           | ~2 G       |\n| Overall                           | 1.2 s + 22 ms  | ——         |\n\n### **Q3: How does GS Fusion differ from earlier multi-query DETR or prompt-fusion approaches**\n\nFor multi-query DETR or prompt-fusion approaches, such as GLIP/GLIPv2 and MI-DETR, they employ cross-modality similarity to update queries one by one or utilize the split-then-concatenate method, whereas VL-SAM-V2 utilizes full similarity to update two queries simultaneously.\n\nWe will add the discussion to the paper."}]

Author rebuttal:

Meta-review: The paper proposes VL-SAM-V2, a unified framework for open-world object detection that integrates open-set (category-guided) and open-ended (category-free) paradigms. The proposed GS Fusion module—enabling synergistic interaction between category-specific queries (from detectors like LLMDet) and general queries (from VLMs like VL-SAM)—demonstrates tangible innovation beyond simple ensemble methods, evidenced by a ​​+2.7 AP gain on rare LVIS objects​ over naive fusion. The framework’s practicality is further strengthened by the denoising point training strategy, which reduces training costs by 50% while maintaining robustness. Post-rebuttal revisions decisively addressed initial reviewer concerns: cross-dataset validation on COCO and CODA  confirmed generalizability, failure cases (e.g., VLM hallucination) were analyzed, and computational overhead was transparently quantified (1.2s inference dominated by VL-SAM). Though the query fusion mechanics draw inspiration from prior attention mechanisms, the holistic integration and state-of-the-art results (41.2 AP rare on LVIS) justify its impact for real-world deployment. 

For camera-ready, authors are encouraged to incorporate the expanded benchmarks and explicitly discuss inference costs in limitations.
Decision: Accept (poster)