Training-Free Open-Ended Object Detection and Segmentation via Attention as Prompts
Authors: Zhiwei Lin
Yongtao Wang
Zhi Tang
Venue: NeurIPS 2024 poster (2024)
Keywords: open-world
open-ended
vision language model
segment anything model
autonomous driving
Primary area: machine_vision
TL;DR: We present VL-SAM, a framework that combines vision-language model with segment-anything model to address the open-ended object detection and segmentation task.

Abstract: Existing perception models achieve great success by learning from large amounts of labeled data, but they still struggle with open-world scenarios. To alleviate this issue, researchers introduce open-set perception tasks to detect or segment unseen objects in the training set. However, these models require predefined object categories as inputs during inference, which are not available in real-world scenarios. Recently, researchers pose a new and more practical problem, i.e., open-ended object detection, which discovers unseen objects without any object categories as inputs. In this paper, we present VL-SAM, a training-free framework that combines the generalized object recognition model (i.e., Vision-Language Model) with the generalized object localization model (i.e., Segment-Anything Model), to address the open-ended object detection and segmentation task. Without additional training, we connect these two generalized models with attention maps as the prompts. Specifically, we design an attention map generation module by employing head aggregation and a regularized attention flow to aggregate and propagate attention maps across all heads and layers in VLM, yielding high-quality attention maps. Then, we iteratively sample positive and negative points from the attention maps with a prompt generation module and send the sampled points to SAM to segment corresponding objects. Experimental results on the long-tail instance segmentation dataset (LVIS) show that our method surpasses the previous open-ended method on the object detection task and can provide additional instance segmentation masks. Besides, VL-SAM achieves favorable performance on the corner case object detection dataset (CODA), demonstrating the effectiveness of VL-SAM in real-world applications. Moreover, VL-SAM exhibits good model generalization that can incorporate various VLMs and SAMs.

Training-Free Open-Ended Object Detection and Segmentation via Attention as Prompts
Zhiwei Lin Yongtao Wang" Zhi Tang
Wangxuan Institute of Computer Technology, Peking University, China {zwlin, wyt, tangzhi] edu.cn
Abstract
Existing perception models achieve great success by learning from large amounts labeled data, but they still struggle with open-world scenarios. To alleviate this issue, researchers introduce open-set perception tasks to detect segment unseen objects in the training set. However, these models require predefined object categories inputs during inference, which are not available in real-world scenarios. Recently, researchers pose new and more practical problem, i.e., open-ended object detection, which discovers unseen objects without any object categories as inputs. In this paper, we present VL-SAM, a training-free framework that combines the generalized object recognition model (i.e.. Vision-Language Model) with the generalized object localization model (i.e., Segment-Anythnn Model), to address the open-ended object detection and segmentation task. Without additional training, we connect these two generalized models with attention maps the prompts. Specifically, we design an attention map generation module by employing head aggregation and regularized attention flow to aggregate and propagate attention maps across all heads and layers VLM, yielding high-quality attention maps. Then, we iteratively sample positive and negative points from the attention maps with a prompt generation module and send the sampled points to SAM to segment corresponding objects. Experimental results the long-tail instance segmentation dataset (LVIS) show that our method surpasses the previous open-ended method on the object detection task and can provide additional instance segmentation masks. Besides, VL-SAM achieves favorable performance on the corner case object detection dataset (CODA), demonstrating the effectiveness of VL-SAM in real-world applications. Moreover, VL-SAM exhibits good model generalization that can incorporate various VLMs and SAMs.
Introduction
Deep learning has achieved remarkable success in perception tasks, with autonomous driving typical practical application. Existing deep learning based perception models rely on extensive labeled training data to learn recognize and locate objects. However, training data cannot cover all types of objects in real-world scenarios. When faced with out-of-distribution objects, existing perception models may fail to recognize and locate objects, which can lead to severe safety issues [24]
Many open-worte world perception methods 15 are proposed to address this issue. Open- perception tries to give precise results in dynamic and unpredictable environments, which contain novel objects and involve scene domain shifting. Current open- world perception methods can be roughly divided into two categories: open-sef and open-ended. Open-set methods [52 43 6 often calculate the similarity between image regions and category names with a pretrained CLIP model. Thus, they require predefined object categories as inputs for the CLIP text encoder during
"ooresponning author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Image Input
Image Feature
Encoder
Decuder
Output Result
Attention Map
Prompt Generation
Prompr
Eacoder
Iterative Refinement
Atteation Map Generation
Answer: On road, there vehicles (including white SUV the foreground), traffic lights, road markings, pedestrian crossing, and pothole Scontinue generation
Question prompt: Uhien-L.angugg Language objects on the road."
Vision-l Model
Illustration of VL-SAM. Without additional training, we connect the vision-language and segment-anything models with attention maps as the intermediate prompts.
Figure 1:
inference. However, in many real-world application scenarios, we do not have the exact predefined object categories. For instance, in autonomous driving, self-driving vehicles may meet unexpected objects, including various rare animals. Besides, some objects cannot be presented by a simple category name, such human in an animal costume, which may look like an animal but is actually a human. Some methods use generic obstacle detection to handle unknown objects. However, many things do not have significant 3D shape, like pits grains the ground. Thus, open-set methods cannot handle all situations. In contrast, open-ended methods [26] 48) are more general and practical since they can predict the object categories and locations themselves.
In a separate line of research, large vision-language models (VLMs) 22
show a strong 23 55
generalized ability to recognize objects, e.g., it can recognize rare objects for corner cases in autonomous driving scenarios [4 However, VLM's localization ability is less accurate than that of specific perception models [51], sometimes missing objects or giving wrong localization results. On the other hand, as a pure vision model, segment-anything model (SAM) [20) exhibits good generalized segmentation capabilities for images from many different domains. However, SAM is unable to provide categories for segmented objects 49 and may yield numerous irrelevant segmentation results.
In this paper, we propose to combine the existing generalized object recognition model, i.e., VLM, with the generalized object localization model, i.e., SAM, to address the open-ended object detection and segmentation task. We present VL-SAM, a training-free framework that connects two generalized models with attention maps the intermediate prompts, as illustrated in Figure I Specifically, we utilize the attention maps generated by VLM when describing the whole driving scene to prompt the segmentation of SAM. Firstly, given the generated token of VLM, use the token the query to obtain the attention maps from all layers and heads of VLM. Then, in the attention map generation module, introduce the head aggregation and attention flow mechanism aggregate and propagate global attention maps through all heads and layers. Besides, to alleviate the collapse problem caused by causal masks when propagating with attention flow, adopt a regularization term constrain the attention flow propagation process. After that, to better guide SAM to segment with the attention maps, we present prompt generation module by grouping and sampling positive and negative points as the point prompts for SAM. Furthermore, to reduce the number of missing objects, we further use the segmentation results from SAM to sample positive and negative points from attention maps iteratively until convergence.
The main contributions of this work are summarized as follows:
We present VL-SAM, training- free open-ended object detection and segmentation frame- work that connects the generalized object recognition model and the generalized object localization model with attention maps as the prompts.
We introduce a head aggregation and regularized attention flow mechanism to aggregate and propagate attention maps with the causal masks through all heads and layers.
We propose an iterative refinement pipeline with a positive and negative point sampling strategy for attention maps.

---

have shown VL-SAM outperforms the open-ended method GenerateU and obtains competitive results compared with existing open-ser methods on the long-tail saigttail instance segmentation dataset LVIS [14]. In autonomous driving applications, VL-SAM achieves favorable corner case object detection performance on the CODA 24
2 Related work
2.1 Vision Language Model
Large language models (LLMs), including GPT-3 [3, GLM
[ and LLaMA 40
human-like dialogue and reasoning skills. However, the limitation of LLM's s ability to process and understand visual data restricts its application more real scenarios. To overcome this, a cutting- edge Vision-Language Model (VLM) is introduced to open up new vistas for application. Recently, BIII-- 23 proposes Q-Former to connect and fuse image and text embeddings with three alignment pretrain losses. I.L.M--Aaapls [53 12], L.LaV, VA (29]. and MiniGPT (55) introduce an adapter or projection layer to align the embedding space from image and text. CogVLMV [4]] presents visual expert modules to transform the image features align with text features in different transformer heads. SPHINX (28) utilizes several mixing techniques for multiple visual tasks. Furthermore, CogAgent [17 and LLaVA-Phi [57] view VLM as an agent assistant to complete various tasks.
Existing VLMs, especially GPT-4V [2] exhibit strong generalization capability for understanding and reasoning new or rare situations, e.g. it can deal with corner cases for autonomous driving [44] However, the localization ability of VI.Ms is weaker than specific perception models, like SAM.
In this paper, we equip VLM with generalized segmentation models, ie., SAM, to address the localization limitation of VLM for open-ended object detection and segmentation. We achieve this by connecting two models with attention maps as the prompts without additional training.
2.2 OpennWorld Object Detection and Segmentation
With the advent of the CLIP models (35), open-world classification, object detection, and instance segmentation have made great progress at the same time. Open-world methods try to discover and recognize unseen objects in the training set during inference. Current open-world methods can be roughly classified into two types: open-set (37] and open-ended [26]. Open-set methods require redefined object categories, including seen objects and unseen objects in the training set, inputs during inference. By contrast, open-ended methods can locate seen and unseen objects and generate their names simultaneously, the current VLM does. In real-world applications, the exact categories may remain unknown for the perception models. For instance, in autonomous driving, self-driving vehicles often encounter unknown objects the road, including overturned cars and construction vehicles with various shapes. Thus, the open- problem is more general and practical.
Open-Set Methods. With the powerful text-image embedding matching with CLIP, current open- set object detection methods mainly use a proposal network to obtain foreground object bounding boxes and embeddings, and then use CLIP as the open-set classification module to predict their categories. More recently, GLIP (5 proposes to use phrase grounding to pre-train open-world object detectors. GroundingDINO 30 presents cross- mods-mtd fusions to introduce text information the image encoder for object grounding. SWORD (55 designs a novel contrastive method to learn the discrimination between foreground and background for instance segmentation. YOLO- (7) introduces prompe-then-detect paradigm for real-time open-world object detection. However, the above methods require predefined object categories as inputs for the text encoder.
Open-Ender Methods. GenerateU 26 first proposes the open-ended problem. Concurrently, DetCLIPv3 [88] introduces similar concept with open-ended. They present generative framework with language models to generate object categories and bounding boxes at the same time. To achieve better generalization capabilities, they construct a large dataset with bounding box and caption pairs and finetune the whole network on the constructed dataset.
In contrast, we propose a training-free open-ended framework, VL-SAM, that combines generalized recognition and segmentation models. VI.-SAM can generate object categories with the generalized recognized model and then localize objects with the generalized segmentation models.
3

---

VL-SAM Framework
VLM
poonger
Prruse
Ouary
Answer: On the noed them vchicls Enchiding white SUY focegroundi, makines,
Teuge
Kiy
Vaice
Atteation
Allcatioe Xap Generation
Hood Aszroaikko
Alteatio Flaws
Mulii-ayer Mulii-hrad Attention
Point Sampling
Refinemen
Oulput Result
Prompt
Encnder
Decoder
Encoder
Paditre Potnr
SAM
X Negudtva Palet
Figure 2: An overview of VL-SAM framework. We first use VLM to describe the input image and generate all possible objects' names. Then, for each object name, we obtain the corresponding attention map with the attention map generation module. Finally, we sample point prompts from the attention map and send them SAM to predict detection and segmentation results.
Method
As shown 2 we provide an overview of our proposed framework. We use VLM and SAM as the generalized object recognition model and object localization model, respectively. Given image input, we first use VLM to describe the scene and list all possible objects in the image. Then, for each object, the attention generation module with head aggregation and attention flow obtain the high-quality attention map from VLM. Finally, we generate point prompts from the attention map and send them to SAM to get the location prediction iteratively.
3.1 Preliminary
Segment Anything Model. SAM is prompt-based segmentation model with excellent data genera- tion capability. It consists of three components: an image encoder, mask decoder, and prompt encoder. SAM takes an image and set of prompts, including points, a box, and mask, as the inputs. To segment objects with the prompts, SAM first extracts image features with the image encoder. Concurrently, the set of prompts is sent to the prompt encoder to transform into the prompt tokens. Then, the image features, prompt tokens, and mask tokens interact in mask decoder with the two-way transformers. Finally, the mask tokens are transformed into multi-scale segmentation masks by multiplying mask tokens with the image features following MaskDINO [22].
Auto-Regressive Based Vision Lasion-Langugge Model. Current Auto-Regressive based VLMs have yielded surprising performance in various vision-language tasks. The mainstream framework of current VLMs comprises four parts, i.e., image encoder, a text tokenizer, projection layers, and language decoder. Given an image and text as inputs, VLMs extract image tokens and text tokens with the image encoder and text tokenizer, respectively. Then, the image tokens are aligned with text tokens with projection layers. After that, the tokens from two modals are concatenated and sent to the language decoder to generate text outputs. The language decoder adopts the next-token prediction paradigm that the probability of the current generated token 2, depends on all previous tokens (æ1, 22, Z(-1).

---

Head Aggregation
Attention Head Weights
MeunMax
W
X
W
Figure 3: Head aggregation. We aggregate information from all attention heads with head weights.
Attention Flow: x 8 x... x
+
Figure 4:
Attention flow. We propagate attention from the first layer to last layer with attention flow.
Question: "List all objects on the roa."
Answer: On the read, there vehicles (a
track and blue hus),
signs,
polesstrian creesing, white burrier, and
other smaller objects that
clearly identifublle from the image.
Question: "List all objects on the road."
the ihere vehicles (including a white SUV the foreground), traffic lights, road markings, pedestrian crossing. and pothole filled asphalt patching material.
Question: "List all objects
Answer: The objects un the road include
white bus, yellow track, traffic cones, a person camoullage
Figure 5: Illustration of attention collapse. For each column, from left to right, we show image inputs, attention flow (collapse), regularized attention flow, and generated answers from VLM.
3.2 Attention Map Generation
The main idea of VL-SAM is to use attention maps of objects as the prompts for SAM to segment. Thus, how to generate a high-quality attention map for an object is critical. To achieve this, we introduce attention flow to aggregate and propagate attention maps through all transformer heads and layers in VLM.
Specifically, given an image input, we ask VLM to give all possible objects in the image. During this process, we cache all queries and keys from VLM. Then, we multiply queries and keys with causal masks and SoftMax normalization to obtain similarity matrix S E N x N x H x L, where N is the length of queries and keys, H is the number of transformer heads, and L denotes the number of transformer layers. shh.! represents the similarity between query i and key j in the head h, layer& 1. After that, we aggregate information from all transformer heads with mean-max attention head weights, of matrix in dimension j and average them dimension obtain the attention head weights W E Elxlx H x
shown in Figure 3 In particular, we choose the maximum similarity weights
W Mean(Max(S, dim = 1), dim = 0).
Obviously, the attention head weight indicates the importance of each head in each layer. Then, we pointwise multiply attention head weight W with similarity matrix S and average all heads as follows:
S' = Me (S & W. dim = 2).
After aggregating information from all heads, we present attention flow to further aggregate attention from all layers, illustrated in Figure Concretely, use the attention rollout method [
5

---

to compute the attentions from layer - 1 to layer as follows:
N
5% = + x (Th. + 5kk)))
where / is the identity matrix, After the attention rollout, we only need the attention map from the last layer. To obtain the image attention map of the generated token, we select the corresponding line and columns from S
However, since VLM uses causal masks for auto-regressive generation, simply adopting the attention rollout method causes attention collapse, shown in Figure [5] Fortunately, find simple regularization term that can alleviate this problem efficiently. Specifically, for each column, assuming the unmasked length is Lo, multiply each value in this column with (Lo - 1)/L. With this regularization term, the attention value in the top left corner will be constrained.
3.3
SAM Prompt Generation
The attention map generated in Section 3.2 has some unstable false positive peaks. To filter these false positive areas, we first use a threshold to filter weak activated areas and find the maximum connectivity area the positive area [5]. The remaining area serves negative area. After that, sample a positive point from the positive area with the maximum activated value and a negative point from the negative area with the weakest activated value. The positive and negative points serve the point prompt pair for SAM.
3.4 Iterative Refinement
The segmentation results from the SAM decoder may include rough edges and background noises. We adopt two iterative strategies to further refine the segmentation results. In the first iterative strategy, we follow the cascaded post-refinement in PerSAM [54] to take the initial segmentation masks generated with the positive and negative pairs as the additional prompt input for the SAM decoder. For the second iterative strategy, to mask the attention map S'. Then, we iteratively generate positive and negative pairs with Prompt Generation in Section 3.3 from the masked attention map and send them the SAM decoder. Finally, we aggregate the results with NMS 13
the segmentation masks in the first iterative strategy
3.5 Multi-scale Ensemble
Due to the low-resolution image input of the image encoder in VLM, VLM may fail recognize small objects. For instance, may generate an answer: 'On the road, there are vehicles (a red truck and a blue bus), road signs, pedestrian crossing, white barrier, and few other smaller objects that are not clearly identifiable from the image.". To alleviate this issue, we follow SPHINX [28] to split image (H x W) into four sub-images (H/2 2 x W/2) from the four corners and send each image to VL-SAM independently. Finally, we ensemble the output of VL-SAM for four sub-images and the whole image.
3.6 Question-prompt Ensemble
The output of VLM sensitive to the input prompt. To obtain a more comprehensive description of the input image, we ask VLM to generate ten question prompts for scene description with the sentence:" If we want you to list all possible objects the given image, what questions should we ask? Please give 10 questions you prefer.' Then, we use the generated question prompts for VL-SAM to segment objects and ensemble the outputs of all question prompts.
Experiments
4.1 Implementation Details
We chose CogVL.M-17 41 with EVA2-CLIP-E [39 and Vicuna-7B-vl.5 [ as the vision-language model. CLL divides image with 490 490 into 35 35 patches. We set the temperature

---

Table 1:
Comparison of object detection and segmentation results on LVIS minival. 'Open-
Ended" denotes that we do not have exact object categories during inference 26 AP 9 for rare objects. denotes using the external data.
We report fixed
LVIS
Method
Type
Training
box AP rrre
mask AP rare
Mask R-CNN [6
26.3
25.1
Close-Set
Deformable DETR
56
24.2
GLIP 25
20.8
GIo
30
27.4
DetCLIP
Open-Set
26.9
YOLOWorld
27.1
OWLv2"
32
39.0
GenerateU [26
V
20.0
Open-Ended
VL-SAM (Ours)
X
23.4
22.7
to 0.8 and top-p for nucleus sampling to 0.1 for Cog CogVLM-17B. For the generated localization model, we use SAM with ViT-Huge [10].
We evaluate VL-SAM a training-free zero-shot manner for all datasets. To obtain object categories from the generated sentence of VLM, we follow Tag2Text [18) parse tags from the given sentence. To evaluate the open-ended performance on datasets with predefined object category names, we follow GenerateU [26] to adopt CLIP [35 text encoder and map the generated object categories to predefined categories in datasets for evaluation. Specifically, we use the text prompt 'a (object category) for CLIP text encoder to calculate the similarity between generated object categories and predefined categories for mapping. All models are inferred on an 80G A800 machine.
4.2 Main Results
LVIS Dataset. We evaluate VL-SAM the LVIS dataset [14), which has long tail of categories and annotations for over 1000 object categories. Following previous works [26 we mainly evaluate VL-SAM on LVIS minival and report the fixed AP ( for rare objects.
As shown in Table I we list the performance for three types perception methods, i.e., close-set, open-set [15]. and open-ended. The different between open-set and open-ended is that open-set requires exact prior knowledge of object categories as inputs, while open-ended can generate them during inference in zero-shot manner (26). In real scenario, we often do not have predefined object categories for a scene. Thus, open-ended methods are more general and practical. As we see, VL-SAM outperforms GenerateU by 3.4 AProre- Notably, VL-SAM is training-fiee framework and can simultancously obtain boxes and segmentation masks. In contrast, GenerateU needs to fine-tune both the image encoder and language model on VG [2I] and GRIT [33 datasets, requiring significant training costs, and can only predict bounding boxes. Besides, VL-SAM achieves competitive detection and segmentation performance compared to open-set detection methods and close-set segmentation methods, respectively.
CODA Dataset. To further demonstrate the effectiveness of the proposed method in the real-world application, we present the results VL-SAM on corner case object detection dataset CODA for autonomous driving in Table 2 Specifically, we can see, RPN only achieves 10.6 mAR, indicating that current open-set detectors relying on object proposals have difficulty dealing with corner cases. For more recent open-set detectors, they achieve higher mAR with CLIP the object category predictor. For the open-ended method, into one model and achieves better performance than open-set methods. However, aggregating VLM and grounding models to one model requires joint training of two models, introducing additional training costs. By contrast, VL-SAM is training- free framework and obtains significant performance improvement over LLaVA LLVAAnGgnn from 18.4 mAR to 40.1 mAR.
V-V-rrunddnnn ensembles VLM and grounding models
In addition, we evaluate the performance upper bound of the current SAM. We utilize ground-truth boxes as the box prompt for SAM decoder to segment objects. We can observe that, this setting, SAM achieves 54.1 mAR and 94.1 AR5o since SAM has its limitations on segmentation tasks.
7

---

Table 2: Comparison of object detection results CODA. We chose the best performance for results from CODA. denotes few-shot object detectors the one-shot setting. 'Oracle' represents utilizing ground-truth boxes as the box prompt for SAM.
CODA
Method
Type
VLM
Training
mAR
AR50
AR75
RetinaNet" 27
X
12.8
23.2
11.9
Faster R-CNN' 36
V
10.7
19.2
10.2
Cascade R-CNN"
X
10.4
18.5
9.7
Deformable DETR"
[56
Close-Set
V
9.0
22.2
5.6
Sparse R-CNN" 38
X
10.1
19.6
9.0
Cascade Swin'
331
9.9
17.2
9.7
RPN 36
X
10.6
20.0
10.2
ORE'
19
8.3
16.4
7.4
FsDer
42
XX
4.2
7.7
4.0
XXX
DeFRCN
34
Open-Set
4.5
8.9
4.2
GNOO
30
12.6
21.7
13.3
YOLOWorld
7
16.1
26.2
19.6
LLa LLaVA-Grounding
[51
18.4
30.5
22.0
Open-Ended
VL-SAM (Ours)
40.1
90.1
50.5
GT+SAM (Oracle)
54.1
94.1
64.9
Table 3: Ablation of main components. 'Attn' is short for 'attention'. Each component improves detection performance consistently.
Naive Attn
Attn Generation
Prompt Generation
Iterative Refine Multi-scale
Question ensemble
mAR
2.2
5.0
10.1
12.3
14.1
27.3
40.1
It sometimes over- under-segments object and cannot obtain perfect segmentation results. Nevertheless, VL-SAM achieves 74.1% 1% mAR performance this upper bound, demonstrating the effectiveness of the proposed framework. Overall, VL-SAM achieves favorable performance on the CODA dataset.
4.3 Ablation Study
Main Components. As shown in Table 3 we conduct ablation studies on CODA to analyze the effectiveness of each component of VL-SAM. For the baseline Naive Attention method, we use the attention map from the last layers and average all attention heads. We can see that the Naive Attention baseline obtains unsatisfactory results even with multi-scale the proposed attention generation module, we improve the baseline by 7.9 mAR. Adding points pairs with prompt generation brings 2.2 mAR improvement. Besides, refining the segmentation maps with the itcrative refinement module improves the detection performance from 12.3 mAR to 14.1 mAR. Furthermore, ensembling with multi-scale image input and question prompt obtains 13.2 mAR and 12.8 mAR, respectively. Though multi-scale and question prompt ensembles greatly improve performance, these two ensemble techniques do not show effectiveness without the proposed components, In summary, the results show the effectiveness of each component proposed in VL-SAM.
question ensemble techniques. With
Attention Generation. obtain high-quality attention maps from VLM, we introduce head weights to fuse transformer heads and regularization term for attention flow. As shown in Table simply using attention flow [ almost fails to recognize objects for SAM due to the attention collapse caused by causal masks (Figure 5). With the regularization term, the attention flow mechanism shows its

---

Table 4: Ablation of attention generation. We can obtain high-quality attention maps with the proposed modules.
Attention Flow
Naive Attention Map
mAR Regularization
Head Weight
No Regularization
2.2
0.1
8.5
10.1
Table 5: Ablation of model generalization. VL-SAM can adopt various vision-language models and segmentation models.
Vision-Language Model
Segmentation Model
mAR
CogVLM
SAM
40.1
MiniGPT-4
SAM
34.7
LLaVA
SAM
37.2
CogVLM
MobileSAM
29.2
superiority over naive attention by improving 6.3 mAR. Moreover, fusing with head weights leads to 1.6 mAR improvement.
Model Generalization. demonstrate the model generalization ability the VL-SAM framework, we adapt two additional popular VLMMs, MiniGPT-4 (55] and LLaVA [29] to replace CogVLM and [50] to replace SAM. In Table we present the results of using these models in the VL-SAM framework. Empirical results show that replacing Cog VLM with MiniGPT-4 or LLaVA may reduce the object localization performance in corner cases as Cog shows more powerful multimodal chat and reasoning ability than MiniGPT-4 and LLaVA. This indicates that VL-SAM framework can benefit from more powerful VLMs. Besides, replacing SAM with a more lightweight but less accurate MobileSAM also leads to performance drops. Nevertheless, all these results outperform previous methods (18.4 4 mAR) in Table 2 This evidences that our framework can generalize to multiple vision-language and segmentation models.
use MobileSAM
5
Limitations
Since we combine VLM and SAM to address the open-ended object detection and segmentation task, VL-SAM inherits the defects of VLM and SAM. The first defect is the hallucination problem in VLM. VI-SAM also suffers from hallucinations, generating wrong object tokens attention maps. The second defect is the low inference speed of VL-SAM. However, these defects can be fixed in the future. For example, there are many more efficient SAM variant models, including EfficientSAN [46] and Mo [50). Our framework can benefit from these new models since easily replace CLg and SAM in VL-SAM with these more efficient and highly accurate models.
Conclusion
In this paper, we introduce VL-SAM, framework that cascades VLM and SAM with the attention map address the open-ended object detection and segmentation task. Without additional training, we adopt attention maps generated by VLM as the prompts for SAM to segment objects. We introduce the attention flow mechanism to aggregate high-quality attention maps. Besides, we present an iterative refinement pipeline with positive and negative points pair sampling strategy to acquire more accurate segmentation masks. Experimental results on the long-tail generic instance segmentation dataset LVIS show that VI-SAM beats the open-ended method GenerateU and achieves competitive performance compared with close-set and open-set methods. Moreover, VL-SAM achieves favorable results on the corner case object detection dataset CODA.

---

Broader Impacts Statement. This paper studies utilizzng VLM and SAM for open-ended object detection and segmentation. We do not see potential privacy-related issues. This study may inspire future research on open-ended perception and potential corner case object detection applications in autonomous driving. However, the proposed model's performance is not yet up to the level of practical application and may pose safety threats when applied directly in practice.
Acknowledgments. This work was supported by National Key R&D Program of China (Grant No.
2022ZD0160305).
10

---

References
[1] Samira Abnar and Willem H. Zuidema. Quantifying attention flow in transformers. In Dan Jurafsky, Joyce Chai, Natalic Schluter, and Jocl R. Tetreault, editors, Annual Meeting of the Association for Compitational Linguistics (ACL), 2020.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint (oXXxx::3330887774 2023.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Neural Information Processing Systems (NeurIPS), 2020.
[4] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. IEEE International Conference on Computer Vision (ICCV), 2021.
[6] Xi Chen, Shuang Li, Ser-Nam Lim, Antonio Torralba, and Hengshuang Zhao. Open-vocabulary panop- tie segmentation with embedding modulation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[7] Tianheng Cheng. Lin Song. Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open open-vocabute vocabulary object detection. arXiv preprint arXiv:2401.17277 2024.
[8] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: open-source chatbot impressing gpt-4 with 90% * chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023), 2023.
[9] Achal Dave, Piotr Dollár, Deva Ramanan, Alexander Kirillov, and Ross Girshick. Evaluating large- vocabulary object detectors: The devil is in the details. arXiv preprint arXiv:2102.01066, 2021.
[10] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai. Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly. Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR. 2020.
[11] Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. GI.M: general language model pretraining with autoregressive blank infilling. In Annual Meeting of the Association for Computational Linguistics (ACL), 2022.
[12] Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model, arXiv preprint arXiv:2304. 15010, 2023.
[13] Ross Girshick. Fast r-cnn. IEEE International Conference on Computer Vision (ICCV), 2015.
[14] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR). 2019.
[15] Akshita Gupta, Sanath Narayan, KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Mubarak Shah. Ow- detr: Open-world detection transformer. In IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[16] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In IEEE International Conference on Computer Vision (ICCV), 2017.
[17] Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang. Zihan Wang. Yuxiao Dong, Ming Ding, el al. Cogagent: A visual language model for gui agents. arXiv preprint arXiv:2312008944 2023.
[18] Xinyu Huang, Youcai Zhang, Jinyu Ma, Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang. Tag2text: Guiding vision-language model via image tagging. arXiv preprint arXiv:2303.05m. 056557, 2023.
[19] KJ Joseph, Salman Khan, Fahad Shahbaz Khan, and Vineeth N Balasubramanian. Towards open world object detection. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021.
11

---

[20] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao. Spencer Whitehead, Alexander C Berg. Wan- Yen Lo, al. Segment anything. In IEEE International Conference on Computer Vision (ICCV), 2023.
[21] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal on Computer Vision (IJCV), 2017.
[22] Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang, Lionel M Ni, and Heung-Youe Young Shum. Mask dino: Towards unified transformer- based framework for object detection and segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[23] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2:ip-2: Bootstrapping language- anguage-imaag pre-training with frozen image encoders and large language models. In International Conference on Machine Learning (ICML). 2023.
[24] Kaican Li, Kai Chen, Haoyu Wang, Lanqing Hong, Chaoqiang Ye, Jianhua Han, Yukuai Chen, Wei Zhang, Chunjing Xu, Dit-Yan Yeung, et al. Coda: A real-world road corner case dataset for object detection in autonomous driving. In European Conference on Computer Vision (ECCV), 2022.
[25] Liunian Harold Li, Pengchuan Zhang. Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[26] Chuang Lin, Yi Jiang, Lizhen Qu, Zehuan Yuan, and Jianfei Cai. Generative region-language pretraining for open-ended object detection. arXiv preprint arXiv:2403. 10191, 2024.
[27] Tsung-Yt Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. IEEE International Conference on Computer Vision (ICCV), 2017.
[28] Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.0757
2023.
[29] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Neural Information Processing Systems (NeurIPS), 2023.
[30] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su. Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.055. 055999 2023.
[31] Ze Liu, Yutong Lin, Yue Cao. Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[32] Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. Neural Information Procexsing Systems (NeurIPS), 2023.
[33] Baolin Peng, Chunyuan Li. Pengcheng He, Michel Galley, and Jianfeng Gao. Instruction tuning with gpt-4, arXiv preprint 2.XXi::334.0332277 2023.
[34] Peter Pinggera, Sebastian Ramos, Stefan Gehrig. Uwe Franke, Carsten Rother, and Rudolf Mester. Lost and found: detecting small road hazards for self-driving vehicles. In International Conference on Intelligent Robots and Systems (IROS), 2016.
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML., 2021.
[36] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection with region proposal networks. Neural Information Processing Systems (NeurIPS), 2015.
[37] Walter Scheirer, Anderson de Rezende Rocha, Archana Sapkota, and Terrance E Boult. Toward open set recognition. JEEE Transactions on Pattern Recognition and Machine Intelligence (PAMI), 2012.
[38] Peize Sun, Rufeng Zhang, Yi Jiang. Tao Kong, Chenfeng Xu, Wei Zhan, Masayoshi Tomizuka, Lei Li, Zehuan Yuan, Changhu Wang, et al. Sparse r-cnn: End-to-eme object detection with learnable proposals. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14454-14463, 2021.
12

---

[39] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint W/Xi:2303.153883 2023.
[40] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anna I Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Erie Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprin! arXiv: 13901,,.111.00000 2023.
[41] Weihan Wang. Qingsong Lv, Wenmeng Yu. Wenyi Hong. Ji Qi. Yan Wang, Junhui Ji, Zhuoyi Yang. Lei Zhao, Xixuan Song, ct al. Cogvim: Visual expert for pretrained language models. arXiv preprint arXiv:331.00377, 2023.
[42] Xin Wang. Thomas E. Huang, Joseph Gonzalez. Trevor Darrell, and Fisher Yu. Frustratingly simple few-shot object detection. In International Conference on Machine Learning (ICML), 2020.
[43] Zhenyu Wang, Yali Li, Xi Chen, Ser-Nam Lim, Antonio Torralba, Hengshuang Zhao, and Shengjin Wang. Detecting everything in the open world: Towards universal object detection. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2023.
[44] Licheng Wen, Xuemeng Yang. Daocheng Fu, Xiaofeng Wang, Pinlong Cai, Xin Li, Tao Ma, Yingxuan Li, Linran Xu, Dengke Shang, et al. On the road with gpt-4v (ision): Early explorations of visual-language model on autonomous driving. arXiv preprint 1rXi::2331.0533. 2023.
[45] Jiannan Wu, Yi Jiang, Bin Yan, Huchuan Lu, Zehuan Yuan, and Ping Luo. Exploring transformers for world instance segmentation. In JEEE International Conference on Computer Vision (ICCV), 2023.
[46] Yunyang Xiong. Bala Varadarajan, Lemeng Wu, Xiaoyu Xiang, Fanyi Xiao, Chenchen Zhu, Xiaoliang Dai, Dilin Wang, Fei Sun, Forrest landola, al. Efficientsam: Leveraged masked image pretraining for efficient segment anything. arXiv preprint arXiv:2312.0086: 2023.
[47] Lewei Yao, Jianhua Han, Youpeng Wen, Xiaodan Liang, Dan Xu, Wei Zhang, Zhenguo Li, Chunjing Xu, and Hang Xu. Detelip: Dietionary-enriceed visual-concept paralleled pre-training for open-world detection. Neural Information Processing Systems (NeurlPS), 2022.
[48] Lewei Yao, Renjic Pi, Jianhua Han, Xiaodan Liang, Hang Xu, Wei Zhang, Zhenguo Li, and Dan Xu. Det- clipy3: Towards versatile generative open-vocabulary object detection. arXiv preprint arXiv:2404.09216 2024.
[49] Haobo Yuan, Xiangtai Li, Chong Zhou, Yining Li, Kai Chen, and Chen Change Loy. Open- vocabulary sam: Segment and recognize twenty-thousan classes interactively. arXiv preprint arXiv:2401.088 022955, 2024.
[50] Chaoning Zhang, Dongshen Han, Yu Qiao, Jung Uk Kim, Sung-Ilo Bae, Seungkyu Lee, and Choong Seon Hong. Faster segment anything: Towards lightweight sam for mobile applications. arXiv preprint arXiv:2306. 14289, 2023.
[51] Hao Zhang, Hongyang Li, Feng Li, Tianhe Ren, Xueyan Zou, Shilong Liu, Shijia Huang, Jianfeng Gao, Lei Zhang, Chunyuan Li, et al. Llava-grounding: Grounded visual chat with large multimodal models. arXiv preprint arXiv:2312.02949, 2023.
[52] Haotian Zhang, Pengchuan Zhang, Xiaowci Hu, Yen-Chun Chen, Liunian Li, Xiyang Dai, Lijuan Wang. Lu Yuan, Lenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. Neural Information Processing Systems (NeurlPS), 2022.
[53] Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. Llama--dapter: Efficient finc-tuning of language models with zero-init attention. arXiv preprint arXiv:2303. 16199, 2023.
[54] Renrui Zhang, Zhengkai Jiang. Ziyu Guo, Shilin Yan, Junting Pan, Xianzheng Ma. Hao Dong, Peng Gao. and Hongsheng Li. Personalize segment anything model with one shot. arXiv preprint arXiv:2305.03045 2023.
[55] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arX1v:2304.10592. 2023.
[56] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable DETR: deformable transformers for end-to-end object detection. In International Conference on Learning Representations (ICLR), 2021.
[57] Yichen Zhu, Minjie Zhu, Ning Liu, Zhicai Ou, Xiaofeng Mou, and Jian Tang. Llava-phi: Efficient multi modal assistant with small language model. arXiv preprint 4rXiv:2401.023300 2024.
13

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility. transparency, research ethics. and societal impact. Do not remove the checklist: The papers not including the checklist will desk rejected. The checklist should follow references and precede the (optional) supplemental material, The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No]. or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant
information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked "[Yes] is generally preferable to "[No] ". is perfectly acceptable to answer "[No] provided a proper justification is given (e.g.. "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset used"). In general, answering "[No] or "[NA] is not phrased in binary way. acknowledge that the answer write a justification elaborate.
the checklist as of the factors in their evaluation. While
grounds for rejection. While the questions often more nuanced, so please just use your best judgment evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can be found.
supporting
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify questions and only use the provided macros for your answers.
Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
We claim the main contribution of this paper in both the Abstract and Introduction
sections.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can he expected to generalize other setrings.
It is fine to include aspirational goals motivation long it is clear that these goals attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of this work in Section S
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged create separate "Limitations" section in their paper.
14

---

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model sell-specification. asympiotic approximations only holding locally). The authors should reflect on how these assumptions might be violated practice and what the implications would be.
The authors should reflect on the scope of the claims made, e-g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicaa,,,, the authors should discuss possible limitations of their approach to address problems of privacy and fairness,
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
Theory Assumptions and Proofs
Question: For each thcoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide the implementation details in Section 4.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
15

---

While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. . a large language model), then there should either be a way to access this model for reproducing the results a way to reproduce the model (e.g.. with open-source datasct instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of booeddsouuce models, it may be that access to the model is limited in some way (e.g.. registered users), but it should be possible for other rescarchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does paper provide open access to the data code. with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: We not provide new datasets and will release the demo after the paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (httpa://nips. httpas///iip..ccppbllleee cc/public/
for more details.
While we encourage the release of code and data, we understand that this might not be possihle, so "No" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines hattpa://nipe http:///nipsccccppllpe cc/public/ htteee//oodSutmiisssooooooo... for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) is recommended, but including URLs data and code permitted.
6. Experimental Setting/Detail
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer. etc.) necessary to understand results?
Answer: [Yes]
Justification: Our method does not require training. We provide the hyperparameters Section 4.1
Guidelincs:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [NA]
Justification: Error bars not reported because our method only involves inference.
Guidelines:
The answer NA means that the paper does not include experiments.
16

---

The authors should answer "Yes" if the results are accompanied by enoor bars, confidence intervals, statistical significance tests, at least for the experiments that support the main claims
the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given
experimental conditions).
method for caleulating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigaa error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be carcful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates). error bars reported tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8, Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: provide information for computer resources in Section 4.1 Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurlPS Code Ethics http::://eerrppp...///l///////be cc/pubblc//hsseeetteee....e
Answer: [Yes]
Justification:
The research in the paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [Yes]
Justification:
We provide the discussion of broader impacts in Section O
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples negative societal impacts include potential malicious or unintended uses (e.g..
disinformation, generating fake profiles, surveillance). fairness considerations (e.g., deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
17

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arisc when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data models that have high risk for misuse (e.g. pretrained language models, image generators, scraped datascts)?
Question:
Answer: [NA]
Justification:
The models in this paper pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All owners of models, code, and data we used are properly cited. We compliance all licenses of models, code, and data.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppappppteeeeco thcode. com/ datasets has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release assets.
18

---

Researchers should communicate the details of the dataset/code/mmoel as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL. or include anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should included the main paper.
According to the NeurlPS Code Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing rescarch with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this the paper.
recognize that procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review.
19


Reviews:
[{"review_id": "xi3GY91EY5", "reviewer": "Reviewer_71UW", "summary": "In this paper, the authors proposed to combine vision language models and segment anything model (SAM) for open-ended object detection. Attention maps generated from vision language models were used to prompt SAM. Experiments on multiple benchmark datasets demonstrated better performance over several baseline methods.", "questions": "Please refer to the weakness section to discuss more on the evaluation and vision language model selection.", "limitations": "The authors discussed the limitations in the draft and I have no further suggestions for improvement.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Combining vision language model and SAM is very interesting and could lead to lots of useful applications. In this paper, the authors proposed an efficient and effective approach to connect them.\n2. The proposed attention map generation from vision language models and prompt generation for SAM make sense and are technically sound to me.\n3. Extensive experiments and ablation studies were conducted to validated the effectiveness of the proposed approach.", "weaknesses": "1. Even though the authors claimed that the proposed approach is open-ended, the evaluation is still on predefined object category names and the performance is worse than some open-set baseline methods.\n2. The performance of the proposed approach heavily depends on the vision language models since the attention map is the key component. Thus, how to choose a good vision language model is essential for the proposed framework. Even though the authors did ablation study of model generation in Table 5, I would expect more discussion on how to choose the best vision language model. Are there any principals or characteristics required?\n3. Several typos: \"visiothe n\" in line 44 and \"Generobtainsnd\" in line 71.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Evaluation is still on predefined object category names.\n\nThe inference of VL-SAM **does not** rely on predefined object category names; we only use them to calculate the evaluation metrics.\n\nSpecifically, as Lines 211-213 mentioned, we use VL-SAM to generate object categories by itself. However, the generated object categories from VL-SAM may not align with the category names specified in the LVIS dataset. For example, VL-SAM may generate \"kids\" or \"adults,\" while the LVIS dataset provides a \"person\" label for these objects. To address this, we follow GenerateU to use the CLIP text encoder to map generated object categories from VL-SAM to the predefined class names in LVIS for mAP evaluation by calculating their similarities.\n\n\n### Q2: Performance is worse than open-set methods.\n\nOur method addresses the **open-ended** detection and segmentation problem, which is different from the **open-set** problem. As mentioned in Lines 34-39, **open-set** methods require predefined object categories as inputs during inference. In contrast, **open-ended** methods do not need to input predefined categories and can predict the object categories themselves. Therefore, it is unfair to compare the performance of our method with open-set methods directly. They are two different tasks.\n\nMoreover, our method is training-free, while open-set methods listed in the paper need additional training.\n\n\n\n### Q3: Vision language model selection.\n\nAs shown in Table 5 of our paper, empirical results demonstrate that VLMs with more powerful multi-modal chat and reasoning capabilities perform better when integrated into VL-SAM. Thus, we suggest using a stronger VLM. \nBesides, our framework is general. Any VLMs that generate object names from images and provide attention maps can be incorporated.\n\n\n\n### Q4: typos. \n\nThanks. We will revise them accordingly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J36loucp2h", "reviewer": "Reviewer_6VtQ", "summary": "This work proposes a novel approach for the so-called open-ended detection (and segmentation) problem, which is about localizing and naming objects in a given image without the user having to specify any pre-defined label space. This is relevant for scenarios where it is hard to define a complete list of object categories that are relevant for the task. For example, autonomous agents acting in the real world may encounter all sorts of objects, but they may not be detected if their category name is missing in the pre-defined label space. The proposed algorithm is interesting because it does not require any model training, but uses readily available vision-language and class-agnostic segmentation models. The experiments also demonstrate strong performance compared to prior works in this setting.", "questions": "- Open Set methods described in lines 104ff are also often referred to as open-vocabulary perception models. I would say more commonly they are referred to as such.\n- Typo in line 153? \"casual\" -> \"causal\"? Same in line 169.\n- The indexing in Eq. 3 regarding the layer index is inconsistent with the above definition.", "limitations": "Limitations of the proposed method were discussed in the main paper. One potential limitation to add would be the inherent two-step approach of the proposed pipeline which cannot recover from mistakes that the initial VLM makes in the captioning output, for instance missing object names.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- Although one might argue that the proposed approach is just a collection of existing models, the proposed combination works, does not require any training, and achieves good results.\n- I think the proposed method can be a very useful tool for many in the research community and in industry. The paper can also be seen as a recipe on how to build similar pipelines for other tasks, without the need to train models.\n- From Section 3 onward, I think the paper was easy to follow. The method is well explained.\n- The ablation study of all individual components is great.", "weaknesses": "- Experiments\n    - The numbers for \"open-set\" (or open-vocabulary) in Table 1 are not state-of-the-art. Looking at OWLv2 [A], even the models not using the LVIS dataset achieve 35.4, 39.0 and 40.9 AP-rare on LVIS-mini with different backbones.\n    - The numbers for non-rare classes in LVIS are not reported in Table 1. Assuming the proposed approach should be a competitive replacement for other object detectors (fixed set, open-set), then I would want to know the numbers on the base classes as well. I expect such a method that relies on attention maps for localization may underperform compared to other methods.\n    - One limitation of the proposed two-step pipeline can be that the initial VLM misses some object names in the caption. The model would not recover even if the segmentation model could segment those objects. Although I saw some Average Recall (AR) numbers in the experiments, I'd be interested in recall of the VLM alone in identifying all object category names that are present. That would have been an easy experiment that should be conducted on standard detection datasets like COCO and LVIS.\n- Paper writing\n    - I had real troubles reading the abstract (and most parts of the introduction as well) and trying to understanding what the task is being solved. The exact definitions of the terms open-world, open-set and open-ended that are used in the abstract may not be clear to everyone. In fact, I think these terms are used inconsistently throughout the ML/CV literature.\n    - The statement that \"pre-defined object categories are not available in real-world scenarios\" probably needs (much) more context to be a valid argument. For instance, autonomous vehicle companies have a list of a few hundred object categories that they expect on the road - hence, it's pre-defined - while unknown objects are handled differently through generic obstacle detection.\n    - The name \"GenerateU\" is used in the abstract without a reference - that's not common knowledge yet - it's a CVPR'24 paper that was recently presented.\n- More details are needed in some parts of the paper:\n    - The reason for collapse of the aggregated attention maps and the corresponding regularization need more details.\n    - Section 3.6 needs more details. To me it seems like the VLM is asked to generate prompts for itself. Is that then dependent on the input image? And what's the prompt to generate more question pairs?\n\nReferences:\n- [A] Scaling Open-Vocabulary Object Detection. Minderer et al. NeurIPS'23", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: The performance of OWLv2.\nThanks for reminding. We will add it to the paper.\n### Q2: Results for non-rare classes in LVIS.\nThe following table provides results for common ($AP_c$) and frequent ($AP_f$) classes of LVIS. We can find that, though Close-set and Open-set methods achieve better results on $AP_c$ and $AP_f$, they require 1203 object names from LVIS as input. Besides, our method is training-free and achieves competitive results with GenerateU.\n|Method|Type|Require Category|Training|Mask|$AP_r$|$AP_c$|$AP_f$|\n|:-|-|-|-|-|-|-|-|\n|Mask R-CNN|Close-set|Yes|Yes|Yes|26.3|34.0|33.9|\n|Deformable DETR|Close-set|Yes|Yes|No|24.2|36.0|38.2|\n|GroundingDINO|Open-set|Yes|Yes|No|18.1|23.3|32.7|\n|DetCLIP|Open-set|Yes|Yes|No|26.9|33.9|36.3|\n|YOLOWorld|Open-set|Yes|Yes|No|27.1|32.8|38.3|\n|GenerateU|Open-ended|No|Yes|No|22.3|25.2|31.4|\n|Ours|Open-ended|No|No|Yes|23.4|25.3|30.0|\n### Q3: Average recall (AR) of VLM for object name generation.\nThe following table provides the AR of VLM for object name generation. $AR_{x}$ presents the similarity score calculated by CLIP larger than x is viewed as positive. \n|Dataset|Prompt Ensemble|$AR_{0.5}$|$AR_{0.7}$|$AR_{0.9}$|\n|:-|-|-|-|-|\n|LVIS|No|97.3|93.2|40.4|\n|LVIS|Yes|98.8|98.4|60.4|\n\n### Q4: Cannot recover missing objects.\nAs shown in the table of Q3, we are currently trying to reduce the number of missing objects with the prompt ensemble. In the future, we will adopt techniques like Chain-of-thought to recover missing objects.\n### Q5: Clarifying the concepts of open-world, open-set, and open-ended perception.\nAs the reviewer mentioned, these three terms are used inconsistently in the ML/CV literature. In our opinion, **open-world perception** is a broad concept. It tries to give precise results in dynamic and unpredictable environments, which contain novel objects and involve scene domain shifting. Open-set and open-ended perception are subtasks of open-world perception, and try to address the novel objects problem. Specifically, **open-set perception**, like the grounding task, predicts object locations when given images and novel object names. In contrast, **open-ended perception** is similar to traditional perception that predicts novel object locations and their names simultaneously when only given images.\nWe will add the discussion to the paper.\n### Q6: Argument of \"predefined object categories are not available in real-world scenarios.\"\nThe reviewer mentioned that  \"autonomous vehicle companies have a list of a few hundred object categories - hence, it's predefined.\" Though they have predefined hundreds of categories, there are still categories they may not include, such as various rare animals. Besides, some objects cannot be presented by a simple category name, such as a human in an animal costume, which may look like an animal but is actually a human.\nAs the reviewer said, generic obstacle detection can handle some unknown objects. However, many things do not have a significant 3D shape, like pits or grains on the ground. Thus, open-set methods cannot handle all situations.\nWe will add the discussion to the paper.\n### Q7: Reason for attention maps collapse when aggregation.\nThe collapse in attention map aggregation is caused by the causal mask. For example, assuming there are uniformly distributed attention maps (3$\\times$3 for simplicity) in all transformer layers (total 2 layers for simplicity):\n$$\n\\\\begin{pmatrix}\n    \\frac{1}{3}&\\frac{1}{3}&\\frac{1}{3}, \\\\\n    \\frac{1}{3}&\\frac{1}{3}&\\frac{1}{3}, \\\\\n    \\frac{1}{3}&\\frac{1}{3}&\\frac{1}{3} \n\\\\end{pmatrix}\n$$\nAfter using the attention rollout method with the causal mask, we can obtain the final attention map:\n$$\n[I+\\begin{pmatrix}\n    \\frac{1}{3} & 0 & 0, \\\\\n    \\frac{1}{3} & \\frac{1}{3} & 0, \\\\\n    \\frac{1}{3} & \\frac{1}{3} & \\frac{1}{3}\n\\end{pmatrix}]\\times[I+\\begin{pmatrix}\n    \\frac{1}{3} & 0 & 0, \\\\\n    \\frac{1}{3} & \\frac{1}{3} & 0, \\\\\n    \\frac{1}{3} & \\frac{1}{3} & \\frac{1}{3}\n\\end{pmatrix}]=I+\\begin{pmatrix}\n    \\frac{7}{9} & 0 & 0, \\\\\n    \\frac{8}{9} & \\frac{7}{9} & 0, \\\\\n    1 & \\frac{8}{9} & \\frac{7}{9}\n\\end{pmatrix}\n$$\nWe can find that, for each row, the number in the front column is greater than in the back column, *e.g.*, $\\frac{8}{9}>\\frac{7}{9}$ and $1>\\frac{8}{9}>\\frac{7}{9}$. Thus, simply adopting attention rollout with the causal mask will centralize the attention activation in the front patch, *i.e.*, the top left corner of the image in order, as shown in Figure 5 in the paper.\n### Q8: Details of regularization.\nAs discussed in Q7, directly adopting attention rollout will make the number in the front column greater than in the back column. We introduce a simple regularization method that multiplies a small term to the front column and a bigger one to the back column. The value of the term uses a simple linear descent: $1-L_0/L$.\n### Q9: Details of prompt generation.\nWe directly ask VLM to generate question prompts by itself without giving images. Specifically, we use the question: \n```\nIf we want you to list all possible objects in the given image, what questions should we ask? Please give 10 questions you prefer.\n```\nHere is the answer from CogVLM:\n```\n1) Please analyze the image and list all the objects present.\n2) Identify and provide a comprehensive list of objects in the image.\n3) Using the image, generate a detailed inventory of all objects visible.\n4) Analyze the image and extract information about any objects present.\n5) Please describe all the objects you can identify in the image.\n6) From the image, generate a report listing all objects detected.\n7) Utilizing the image, identify and report all objects visible.\n8) Analyze the image and provide a comprehensive breakdown of all objects seen.\n9) Please process the image and generate a detailed list of all objects present.\n10) Using the image, identify and describe all objects observed.\n```\nWe will add the details in the paper.\n### Q10: Typos and indexing inconsistency.\nThanks. We will revise them in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P9ZiIxMJqc", "reviewer": "Reviewer_bgFd", "summary": "This paper proposes a training-free open-ended object detector that leverages VLM(CogVLM) to recognize and roughly locate objects(through the attention map) and prompts SAM by the coarse points results. To generate accurate point prompts, VL-SAM utilizes techniques like head aggregation, attention flow, attention score regularization, iterative refinement, and scale/prompt ensemble. VL-SAM achieves state-of-the-art performance on open-ended LVIS and CODA. The ablation proves that each component can improve the detection performance.", "questions": "1. types: line 44 as a pure visiothe n model\n2. What is the question-prompt, and how does it affect the final performance?\n3. How much would VL-SAM's performance be affected by the model size?\n4. see weakness.", "limitations": "yes, the authors has adequately discussed the potential negative societal impact of their work and limitations.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "Combining already trained large models to achieve tasks that each cannot accomplish individually is a promising direction. The techniques proposed by the authors—head aggregation, attention flow, attention score regularization, iterative refinement, and scale/prompt ensemble—are all intuitively effective methods for improving zero-shot performance.", "weaknesses": "1. Speed. For each image, following the VL-SAM approach requires first passing through CogVLM-17B ten times, with each pass covering five scales (full image + 4 sub-images). Afterward, attention needs to be stored and calculated to obtain the prompt. Each prompt then requires iterative refinement, with each refinement step involving multiple passes through the SAM-huge model. Each step may also require post-processing like NMS. Such heavy and non-parallelizable computation raises curiosity about the model's FPS. Is the performance gain really worth it?\n\n2. Unclear experimental details. Table 3 indicates that the highest improvements come from question ensemble and multi-scale operations. Firstly, these operations cannot be considered core contributions of VL-SAM, as they are widely used tricks. Secondly, examples of question prompts are not provided. Given that question ensemble improves performance significantly, additional analysis is required to understand why.\n\n3. SOTA performance. On the LVIS minival dataset, the authors primarily compare against GenerateU, as mentioned on line 114. DetClipV3 also proposed an open-ended setting and achieved higher performance, yet this is not reflected in Table 1. Additionally, it should be noted that GenerateU used different types of CLIP text encoders for training and evaluation. In my attempt to unify both for retraining, I achieved approximately 25 box AP on rare classes, which is close to the 23.4 performance mentioned in the paper (though this will not affect my scoring, as it is not an issue with VL-SAM). On the CODA dataset, VL-SAM achieved performance far exceeding other methods. However, considering that the evaluation metric is AR, a natural question arises: what would be the AR if SAM's segment-anything mode (uniform sampling) was used? This is necessary to demonstrate the need for VL-SAM. Additionally, in Table 3, VL-SAM without the two ensemble methods only achieves 14.1 performance on CODA, while using both methods introduces significant speed overhead (see point 1).\n\n4. Parameter: VL-SAM defaults to using some of the largest foundation models (SAM-Huge, ViT-E). Is the comparison of VL-SAM's parameter count to other methods fair? Additionally, what would VL-SAM's performance be if smaller-scale models were used?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Q1: Speed and parameter.\n\nAs we discussed in the limitation, the speed and parameter problem can be gradually overcome by recent lightweight models. Besides, the main purpose of this paper is to provide a **feasible solution** to address the open-ended perception challenge; thus we do not consider the speed and parameters right now.\n\nAdditionally, we analyze the effect of SAM‘s parameters on VL-SAM, as shown in the following table. We can find that a larger model achieves better performance for VL-SAM. Besides, when parameters are reduced from 636 M to 91 M of SAM backbone, the performance of VL-SAM only drops 2.2 mAR. This demonstrates the effectiveness of the proposed framework once again.\n\n| SAM backbone | Parameters | CODA mAR |\n| :----------- | ---------- | -------- |\n| VIT-H        | 636 M      | 40.1     |\n| VIT-L        | 308 M      | 39.3     |\n| VIT-B        | 91 M       | 37.9     |\n\n### Q2: Details and analysis of prompt generation.\n\nWe directly ask VLM to generate question prompts by itself without given images. Specifically, we use the question: \n\n```\nIf we want you to list all possible objects in the given image, what questions should we ask? Please give 10 questions you prefer.\n```\n\nHere is the answer from CogVLM:\n\n```\n(1) Please analyze the image and list all the objects present.\n(2) Identify and provide a comprehensive list of objects in the image.\n(3) Using the image, generate a detailed inventory of all objects visible.\n(4) Analyze the image and extract information about any objects present.\n(5) Please describe all the objects you can identify in the image.\n(6) From the image, generate a report listing all objects detected.\n(7) Utilizing the image, identify and report all objects visible.\n(8) Analyze the image and provide a comprehensive breakdown of all objects seen.\n(9) Please process the image and generate a detailed list of all objects present.\n(10) Using the image, identify and describe all objects observed.\n```\n\nMoreover, as Reviewer 6VtQ mentioned, we provide the average recall (AR) of VLM for object name generation in the following table. We find that prompt ensemble significantly alleviates the object missing problem in VLM. Thus, it can improve overall performance.\n\n| Dataset | Prompt Ensemble | $AR_{0.5}$ | $AR_{0.7}$ | $AR_{0.9}$ |\n| :------ | --------------- | ---------- | ---------- | ---------- |\n| LVIS    | $\\times$        | 0.973      | 0.932      | 0.404      |\n| LVIS    | $\\checkmark$    | 0.988      | 0.984      | 0.604      |\n\nWe will add the discussion to the paper.\n\n\n\n### Q3: AR of SAM's segment-anything mode (uniform sampling).\n\nWe use SAM's segment-anything mode to obtain box proposals and calculate the class-agnostic AR. The AR is 29.7. However, the boxes predicted by SAM do not contain category results. In contrast, VL-SAM can predict boxes with categories.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their constructive comments. They recognize that our work can "lead to lots of useful applications" (71UW, 6VtQ, bgFd), "achieves good results" (71UW, 6VtQ), and is "well explained" (71UW, 6VtQ) and "effective" (71UW, bgFd). We address the reviewers' concerns in the rebuttal text below.

Meta-review: The paper received one Borderline Reject and two Borderline Accepts. All reviewers agree that the paper introduces a novel system to combine a VLM with SAM for open-ended object detection. The method adopts the attention map in a VLM as a localized prompt to SAM for object segmentation. 6VtQ and 71UW agree that the combination of a VLM with SAM cloud leads to many useful applications. The main concern of bgFd is that the proposed model, while novel, may not be suitable for real world applications due to its latency on iterative refinements. After considering the review, rebuttal, and manuscript, AC agrees that the proposed method could inspire future works on grounding text outputs of a VLM to localized regions in a training-free way. The novelty outweighs the practical consideration raised by bgFd for deploying the method in real-time applications. Consequently, AC recommends accepting the paper.
Decision: Accept (poster)