Fantastic Tractor-Dogs and How Not to Find Them With Open-Vocabulary Detectors
Authors: Frank Ruis
Gertjan J. Burghouts
Hugo Kuijf
Venue: ICLR 2026 Poster (2026)
Keywords: open-vocabulary
object detection
vision-language
false positives
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We show that open-vocabulary detectors significantly degrade on images where no prompted object is visible, and propose a simple training-free fix.

Abstract: Open-Vocabulary Detectors (OVDs) excel in zero-shot benchmarks, but we observe a critical flaw in real-world deployment: a high rate of confident false positive predictions on images that do not contain any target objects (e.g., detecting a tractor in an image of a dog). This issue is masked by standard benchmarks like COCO and LVIS, as they rarely contain images without any of the target classes present. We identify vision-language fusion layers in early-fusion OVD architectures (e.g., Grounding DINO or LLMDet) as the root cause, and show how they distribute irrelevant class information across image features when no prompted object is present. To mitigate background false positives without costly retraining, we propose a simple, training-free method: appending attention sink tokens to the input prompt. We show that such sinks can redirect spurious attention and dramatically reduce background false positives. Our approach significantly improves the performance of all six early-fusion models tested (e.g., boosting AP on LVIS by more than 5x at a false positive rate of 0.01 for some models), making them practical for real-world applications where images without the object of interest are much more prevalent.

Published as a conference paper at ICLR 2026
FANTASTIC TRACTOR-DOGS AND How NOT TO FIND THEM WITH OPEN-VOCABULARY DETECTORS
Frank Ruis, Gertjan J. Burghouts, & Hugo Kuijf
TNO, Intelligent Imaging frank. ruis@tno...
ABSTRACT
Detectors (OVDs) excel in zero-shot benchmarks, but we ob- serve critical flaw in real-world deployment: high rate of confident false posi- tive predictions on images that do not contain any target objects (e.g., detecting a tractor an image of a dog). This issue is masked by standard benchmarks like COCO and LVIS, as they rarely contain images without any of the target classes present. We identify vision-language fusion layers in early-fusion OVD architec- tures (e.g., Grounding DINO or LLMDet) as the root cause, and show how they distribute irrelevant class information across image features when no prompted object is present. To mitigate background false positives without costly retraining, we propose a simple, training-free free method: appending attention sink tokens to the input prompt. We show that such sinks can redirect spurious attention and dra- matically reduce background false positives. Our approach significantly improves the performance of all six carly-fusion models tested (e.g., boosting AP on LVIS by more than 5x at a false positive rate of 0.01 for some models), making them practical for real-world applications where images without the object of interest are much more prevalent.
INTRODUCTION
Opea-locabulary Detection (OVD) models boast increasingly impressive performance on zero-shot benchmarks (Liu al.] 2024b Ren et al. (2024b] Minderer et al. (023). but their propensity for false positive predictions severely limits their applicability outside academic contexts. An exam- ple of this is shown Figure D where any prompted class or phrase will result in confident false positive predictions on semantically irrelevant objects. While the literature on
Model (VLM) ballucination (Sarkar et al.) 2025] Yang et al., 2025 al.) |2225
al.. 2025)
is steadily growing, there is notable lack of similar studies focused on OVD models. Although the issue has been noticed for quite some time by practitioners (GitHub Issue 2023). it remains largely unexplored in academia, apart from brief mention in the Grounding DINO I.5 Technical Report (Ren et al., 2024b), highlighting the increased use of negative samples during pre-training. One possible reason for this gap is that false positives in OVD models tend occur primarily in background-only images, where none of the target classes are present. However, standard training datasets and benchmarks such COCO (Lin et al.) 2015)) and LVIS (Gupta al. 2019) almost al- ways include at least one annotated ground truth instance per image. This leads to strong benchmark performance, yet fails to reflect many real-world scenarios where background images are far more common than those containing foreground instances, such as in security or medical imaging.
We investigate the cause of this background false positive rate (FPRgg) and highlight the differ- ences between carly-fusion and late-interaction OVD models with respect to FPRpg- Specifically, we quantitatively show that only early-fusion models suffer from high FPR compared to their FPR on foreground images. For late- iate-interaction models, there is no significant deviation between the two. Because early fusion models offer strong benefits over late interaction models in certain complex tasks, such as visual question answering (Marino et al. 2019 Goyal et al. 2017) and refer- ring expression comprehension (Kazemzadeh et al. [014 4 He et al. 2023). it would be beneficial to mitigate their high FPRpg We establish that cross modal attention operations in the rision-language fusion layers of carly-fusion models are the cause of the high FPRgg and propose a training -feee attention sink approach to almost completely mitigate the issue. This finding not only reveals

---

Published as a conference paper at ICLR 2026
bocter according to ol knowen laws of oxtation therels 1he wow bee shezul lyy.
TIDEDO gross.
Figure 1: Example of Grounding DINO (Liu et al. 2024b) confidently predicting "tractor" on picture of a golden retriever. We show that this is a common error for open-vocabulary detectors when prompting for object that does not occur in the image.
previously overlooked limitation in the architectural design of early- OVD models, but also opens up broader line of inquiry into how rross-modal attention mechanisms be rethought to balance generalisation with reliability in real-world deployments.
Our main contributions be summarised as follows:
We identify a pervasive issue of confident false positive predictions across 4 state-of-the- art vyen-vocabulary object detectors, significantly impacting their practical usability, and show why this has not been identified by any commonly used object detection benchmarks.
We propose simple adaptation to existing benchmarks to identify and quantify background-only false positives.
We show that fate-interaction models do not behave differently background images, while early-fusion models do, and offer a potential explanation of its cause through in- specting the attention patterns in vision-language fusion layers.
We propose training-free approach to mitigate hackground-only false positives: simply appending attention sink tokens to the prompt reroute attention away from negative classes. This results in significant improvements in false-positive mitigation with minimal impact on the performance on positive samples.
PRELIMINARY: EARLY FUSION AND LATE INTERACTION
OVD models can be broadly categorised early fusion or late interaction, depending on how and which point in the architecture the vision and language modalities interact (cf. Figure 2}. As name implies, early fusion models, such GLIP (Li et al. 2022) and Grounding DINO (Liu et al., 2024b), combine vision and language features, usually through a cross-attention operation. followed by further processing by additional modules. Late interaction models, such as CLIP (Radford al. 2021) or OWL-ViT (Minderer et al. 2022), never fuse vision and language features but align the feature outputs of two modality-specht encoders through vector similarity metric.
Early Fusion
Late Interoction
moce
"Dog."
"Dog""
bachete
"Dog
Poce
Figure 2: In early fusion models the vision and language features are combined, usually through cross-attention operation, followed by further processing by additional modules. Late interaction models never fuse features, but align the feature outputs of two -opelity-specific encoders through vector similarity metric.
2

---

Published as a conference paper at ICLR 2026
Although late interaction performs well for tasks such retrieval and regular object detection (Cheng et al.; 2024, Minderer et al. 2023). the lack of cross cross-modal interaction means it cannot effectively perform more complex tasks such referring expression comprehension (REC) vi- sual question answering (VQA) (Shin et al. 2022 Alberti et al. 2019). The current state-of-the-art (Ren et al. 2024a) suggests that early fusion methods also outperform late interaction methods in lasks such as regular object detection; however, in Section 3 we show that this is not necessarily the case.
IDENTIFYING THE FALSE POSITIVE PROBLEM
Quantification As mentioned in the Introduction
! standard benchmarks such as COCO (in
et at.l. and LVIS (Gupta et al. (2119) mask the issue of background-only false positives because they rarely contain background images. Even they do, common training configurations popular frameworks such as mmdetection (Chen et al. (019) are set up to filter out images that do not contain ground truth boxes. L.VIS is a federated dataset, which means that its classes are not exhaustively labelled for every image. Instead, each individual image has set of positive classes that have been labelled, as well as a set of negative classes that have been confirmed to not be visible in the image. We leverage these negative labels to quantify the problem; by prompting the model with exclusively negative classes, we know that any prediction above a given confidence threshold can be considered a false positive.
For each image, we loop over all positive and negative labels and prompt the OVD model one class time. The predictions positive labels are used to calculate the average precision (AP) as usual. The predictions of the negative labels are used to determine the number of false positives per background image. which we denote as FPRag. The FPRgg can be decreased by increasing the confidence threshold for accepted predictions. We calculate an AP at specific FPRng value of fpr by increasing the confidence threshold for predictions until the desired FPRng value is reached. Once the desired FPRby is reached, we filter out positive predictions below the new confidence threshold and calculate the corresponding AP.
Early fusion models degrade on background images Figure 3 shows the LVIS MiniVal AP of eight OVD models compared to their AP Ideally, models that do not behave differently on background-only images should lie along the diagonal, indicating no drop in performance. The vertical deviation below the diagonal signifies the model's propensity towards background-only false positives. We can see clear divide between late-interaction and early-fusion models: Late- interaction models behave similarly to regular (non-OVD) models in that they may have some false positives background images, but not to the extent that it significantly impacts their performance on positive samples when picking a confidence threshold. Early fusion models, on the other hand, experience a significant drop in performance on positive samples when lower their FPR
Why not use only late interaction architectures? A simple solution then seems to be to just not use early fusion, which may be valid in certain use-cases. However, early-fusion models offer several key advantages over late- iite-interaction models, such as improved compositional and out-of- domain generalisation (Zhou et al., 2022), and the ability to perform more complex tasks such as Referring Expression Comprehension (REC) or Visual Question Answering (VQA), which are ill- suited for ineeinteraction architectures (Shin et al. 2022 Alberti et al. 2019).
Earlier solution attempts are not scalable To increase robustness to FPRpg- the authors of Grounding DINO, for their subsequent closed source models, sample significant amounts of ad- ditional negative examples during pre-training, as mentioned in the Grounding DINO 1.5 technical report (Ren et al., 2024b). However, this approach does not seem scalable, given that it remains exceedingly easy to find irrelevant classes with confident false positive predictions even for their most potent models (e.g. red panda being detected as jenga blocks, cf. Figure [0 in the Appendix). may be hard or even impossible to train on sufficient negative examples, due to the combinatorial explosion of possible negative classes and objects in images.
There is no "None of the "bove" Clearly, early- fusion models are capable of recognising that a dog is in fact not tractor, as evidenced by the fact that the mistake usually disappears when adding
3

---

Published as a conference paper at ICLR 2026
False Positive Mitigation with Attention Sinks
Farte Rusion
Late Interaction
Attention Sinks
Froe madet apis Seport bet Torencee
interoncus DelInterrmeus (modale" cEa, mrdeii-pp py"
woligtt si pebit)
nst
1e" [unusod[1]]
rango [nel)]
inferenazz model, Lanquage, podel, cokenizez cdd tokenn (o))
AP615
inforencez. model
[[lcnnit1] K.neur (dimel)) unuqquesse (0), repeatt (nxi, 1)
20
(*Lractoo")
Lndercnoee (90", sbow True, texte
10-
(ilowr out prodiet lono iniing alaka
keep to (clo) for e pol [01 ("Iabola"13 (kk foz i, (k. in caunerate teo. .ionno xeop
20 30
LVIS VaniVal AP
Figure 3: On the left, show the LVIS Mini AP compared to the AP at an FPRag of 0.05 for 8 open-vocabulary detectors, in a zero-shot setting. While most models perform worse when taking background images into account (below the diagonal), the performance drop is especially severe for all early fusion models (lower right). Adding attention sinks (+) significantly improves performance for all tested models. On the right, stand-alone snippet of code showing the case of implementing attention sinks for MM-Grounding DINO.
different positive class to the prompt. Inspecting the attention patterns of vision-language fusion layers, we find that the issue likely stems from the nature of the attention operation not being able to pick zero tokens in the absence of good match. Figure shows an example of such visualisation for positive, negative, and positive + negative prompt. For the negative prompt tractor, every vision token incorporates at least some information of the tractor class since there is other prompt token to choose from. However, when adding the positive class grass, the vision tokens containing grass can instead attend the grass class, while the other vision tokens can incorporate a mix of both classes. This seems to be enough for the model to calibrate its confidence so that only grass will be detected. Adding one or more additional negative classes to the prompt does not mitigate the number of false positives for background-only images, the most "prevalent" class features in the vision tokens will always get a high confidence score. We hypothesise that in later stage the model calibrates confidence scores according to the relative proportions of class information incorporated by each of the vision tokens, such as in the grass example. We observe that, in the absence of a clear positive class, of the negative classes will be picked instead, leading confident false positives.
Dog
Tractor
Tractor. Grass
Figure 4: A visualisation of the attention scores of each head (horizontal) and scale (vertical) from the first vision- vision-language fusion layer of LLMDet (Fu al., 2025), between the visual features of an image of golden retriever and the prompt tokens "dog" (left) and "tractor" (middle and right). When prompted for tractor (middle), significant attention routed to the visual features of a dog, causing the classification head to predict the tractor class when presented with the class-agnostic object query containing the dog features mixed with the tractor token. Adding a positive class (grass, right) partly mitigates that, due to the model being better at calibrating confidence scores when a positive class is present.
4

---

Published as a conference paper at ICLR 2026
4 REDISTRIBUTING IRRELEVANT INFORMATION TO ATTENTION SINKS
Recent work in both language and vision has shown that attention sinks (iiao al. 2023). or reg- ister tokens (Darcet et al. 2023). may be needed to prevent transformer-base models from reusing certain input tokens as scratch pads for unrelated computations. purpose low-information tokens to store global information. While this mechanism is helpful for achieving strong overall performance, it disrupts locality of information and can hurt performance on dense prediction tasks (Darcet et al.) 2202)). Interestingly, this behaviour differs from what observe in the vision-lane -aiiunge fusion layers of OVD models (Section 3 Instead of concentrating important information into few tokens, these models spread irrelevant class information across many tokens.
practice, these models often re-
This diffusion of irrelevant information is consistent with how such models are trained. Since ery image always contains at least one positive class, and irrelevant information cannot simply be discarded, the model only needs to dilute irrelevant signals enough so that they remain below the confidence threshold of the positive class in the prediction heads. However, when no positive class is present, this irrelevant information becomes the dominant class signal in the features. Luckily, this does suggest that the model has already learned to minimize the presence of irrelevant information in its features. To prevent false positives, we only need allow the model pick features that do not pertain to any class in particular.
Here where attention sinks can serve useful role, specifically, as the missing "none of the above" option discussed in the previous section. By introducing one or more semantically neutral sink to- kens into the input prompt and treating them as regular target classes, the model naturally routes excess or irrelevant attention to these sinks. When the model assigns a sink token as the predicted class for an object, that prediction can safely be discarded (cf. Figure 5p. This approach reduces background false positives while preserving the model's calibration: positive class predictions main largely unaffected even after sink tokens are integrated into the feature map.
103913
0.99
0.99
0.84
0.02
Text
0.01
0.16
0.98
Model Outpets
Creis Modally
Lingeage
Ducry Selection
Tel
Erd
Trecttc
Vandle tver
Verlls Image
"tat
sooibore
Dog Tractor
Input
Ingut Image
Figure 5: A diagram showing an example attention patterns for queries containing a positive, only negative, and negative and attention sink tokens respectively. Modification of an image taken from Grounding DINO (Liu et al. 2024b).
To initialise the attention sink tokens s,, we use one of three initialisation strategies:
NO Σ
22)
(random init)
8:
IV
cw
(mean of all word embeddings e., in vocabulary V) for i 1, 1,..., Nsinks;
WEV
e
(embedding of checial-character string "[O]")
5

---

Published as a conference paper at ICLR 2026
We find that different initialisations work better for different models and elaborate choices in Appendix A.2] The attention sinks work out-of-the- with all models evaluated in the paper, without the need for any training.
In addition to language sinks in the prompt, we also experimented with vision sinks by introducing extra sink tokens into the vision features every language- fusion step. While this alone can improve performance, we observed that false positives are primarily driven by the vision-to- -onggugge attention direction. With well-chosen language sink initialisations, the need for vision sinks largely disappears. This suggests that the effect of irrelevant label information in vision tokens is minimal, likely due the large number of vision tokens available, and their existing integration of language sink features when also using language sinks.
5 EXPERIMENTAL SETTINGS
We study how well attention sinks can mitigate background false positives (FPRig) on common OVD benchmarks, ideally without affecting the performance on positive classes. Attention sinks can be implemented in just a couple of lines of code, an example of which can be seen in Appendix
A.
Benchmarks The POPE benchmark (Li et al. 2023 probes large vision-language models for object hallucinations. It uses images and annotations from COCO (Lin al. 2015), with binary labels that denote the presence of given object. These are used to evaluate vision-languae models with language outputs on yes-no questions. We convert POPE back the object detection task by using the ground-truth bounding boxes of given POPE object and image instead of a binary label, and use the standard average precision (AP) metric over model predictions. For negative classes, we record all predictions and their confidence scores to determine the FPR at a given confidence threshold. An added benefit of POPE's approach of probing for one object at a time that this eliminates the large variance in prediction results for early-fusion models when changing the order or number of classes in the prompt.
The experiments for other datasets are constructed analogously to POPE. LVIS (Gupta et al. 2019) is the main benchmark used to evaluate the zero-shot generalisation for opca-vocyy object detectors. Because LVIS is federated dataset, each image instead. The goal is to reach a high AP while keeping FPR low. The FPR is defined as the number of false positives per prompt, i.e. an FPR of 0.75 on LVIS indicates that you can expect any of the 1203 LVIS classes to result in false positive prediction on 75% of images that do not contain that class.
the explicit negative classes provided for
In the following sections, we focus on the LVIS dataset because it is the most comprehensive and widely used dataset for opeabulcybulary detection. Results of additional experiments, including POPE, can be found in Appendix A.3
Models We evaluate five early fusion models, GLIP (Li et al. 2022). OmDet-Turbo (Zhao et al. 2024a), Grounding DINO (Liu et al. 2024b), MM-Grounding DINO (Zhao et al.] (024b), and LLMDet (Fu et al. 2025). and three late interaction models, YOLO- World (Cheng et al. 2024). OV-DINO (Wang et 2024bb, and OWL-ViTv2 (Minderer et al. 2023)
We explore the optimal number of attention sinks for various models as well as the type of initialisation of the attention sinks (random, mean, specific words or tokens). We also explored learning the word embedding weights of the attention sinks, which we elaborate on in Appendix AA44, but that did not offer improvements over the training-free approach. Ablations are performed on 64 images of the VOC dataset, with two positive and two negative prompts per image. We combined the attention sink initialisation experiment with search of the minimum number of images from the 22-class VOC dataset that can offer a stable performance indicator of the more complex 1203-class LVIS dataset. We did so by first searching over a range of 8, 16, 32, 64, 128. 256. 512. and 1024 images find the minimum number of VOC images that maintains a stable performance over different subsets of images and then compared that performance to the same attention sink initialisation the LVIS Mini dataset.
Attention Sinks

---

Published as a conference paper at ICLR 2026
6 ANALYSES AND FINDINGS
Can attention sinks reduce background false positives? In Figure we showed that early fu- sion models experience a sharp performance drop on LVIS once background false positives are considered. For example, on LVIS MiniVal, LLMDet Swin-B achieves AP of 0.466. However, this comes with an bg of 2.4 at a confidence threshold of 0.25, i.e., more than two false positives per image and negative prompt average.
Table T further details the LVIS ViniVal results of eight OVD models at five different FPRgg val- ues, without and with our attention sinks. Attention sinks aim keep model performance low FPRig close to the standard AP (final column), which would indicate full mitigation of background false positives without hurting positive-class detection. The bottom section of Table T shows early- fusion models augmented with attention sinks, with absolute improvements highlighted in green. At low Fg all models see large AP gains. While late- interinttraction models still achieve the best performance at near-zero false- false-positive rates, attention sinks make early-fusion models competitive at practical error levels. For example, L.LMeet- more than doubles its AP at FpR 0.05 and achieves five-fold increase FPRgg 0.01. Some models show slight drops in AP when ignoring background false positives (final column). largely due to low-confidence predictions falling below the threshold after attention sinks are applied (see Figure 6 for an example). Crucially, almost all background false positives disappear. The few that remain are similar to the errors seen in late- interaction models, such as fine-grade tineinnained errors (e.g., Labrador detected Cocker Spaniel) or visual similarity errors (e.g., regular pumpkin detected as a jack-o' -lantern).
Model
APERR
AP 0.05
AP
AP
AP
Early-Fusion
OmDet Turbo
0.035
0.075
0.093
0.127
0.228
GLIP Swin-T
0.116
0.180
0.221
0.257
0.333
GDINO Swin-T
0.037
0.098
0.141
0.211
0.396
MM-GDINO Swin-T
0.076
0,148
0.184
0.257
0,407
LLMDet T
0.045
0.140
0.195
0.258
0.464
LLMDet Swin-B
0.047
0.125
0.166
0.218
0,466
LLMDet Swin-L
0.113
0.218
0.263
0.340
0.488
I LtteInteraction
YOLO-World L.
0.189
0.238
0.245
0.245
0.245
OV-DINO Swin-T
0.274
0.316
0.325
0.336
0.349
Carly-Fusion attn sinks
GLIP Swin-T
0.166 (+0.050) 0.229 (+0.049)
0.256 (+0.035)
0.283 (+0.026)
0.302 (-0.031)
GDINO Swin-T
0.073 (+0.036)
0.152 (+0.054)
0.192 (+0.051)
0.256 (+0.045)
0.359 (-0.037)
MM-GDINO Swin-T
0.214 (+0.138) 0.306 (+0.158)
0.348 (+0.164)
0.394 (+0.137)
0.426 (+0.019)
LLMDet Swin-T
0.223 (+0.178) 0.326 (+0.186) 0.359 (+0.164)
0.400 (+0.142)
0.448 (-0.016)
LLMDet Swin-B
0.222 (+0.175) 0.349 (+0.224) 0.403 (+0.237) 0.449 (+0.231)
0.499 (+0.033)
LLMDet Swin-L
0.182 (+0.069) 0.318 (+0.100) 0.387 (+0.124) 0.442 (+0.102) 0.502 (+0.014)
Table 1: Results on our proposed LVIS hallucination detection benchmark, at five levels of FPRg Although late-interaction models still outperform early-fusion models near-zero false positive rates, attention sinks make early-fusion models competitive again at manageable false positive rates.
What is the effect of attention sinks on the vision-language fusion layers? Figure visualises the attention scores from the first vision-language fusion layer of LLMDet-T. Without attention sinks, almost all visual features attend to the negative class token tractor, which confuses the clas- sification head. Adding attention sinks redirects this spurious attention to sink tokens, leaving the tractor token with the desired near-zero attention. For the true positive class dog. the attention map also becomes much cleaner: background tokens no longer absorb information from the dog token. As result, the classification head avoids the calibration issues seen without attention sinks.
7

---

Published as a conference paper at ICLR 2026
No Sinks
With Sinks
No Sinks
With Sinks
S0
Bicycle
Bicycl
Parokeet
Porokeet
SE
Bread
Brood
Figure 6: Qualitative examples of zero-shot predictions with LLMDDt Swin-T LVIS MiniVal images, with and without using attention sinks. Without attention sinks, the model has many false positive predictions regardless of semantic relevance, e.g. misidentifying skis as a parakeet. While in some cases low-confidence true positives will become false negatives when using attention sinks (e.g. one of the bicycles), almost all cases of background false positives are mitigated.
7
Dog
Dog attn sinks
Tractor
Tractor attn sinks
Figure 7: A visualisation of the attention scores of each head (horizontal) and scale (vertical) from the first vision- saiinnla fusion layer of LLMDet, between the visual features of an image of golden retriever and the prompt tokens "dog" (top) and "tractor" (bottom), without using attention sinks (left) and with attention sinks (right). Both the positive and negative class have a much cleaner attention map after adding attention sinks, with most irrelevant information being routed away from negative classes to the attention sinks.
How do find the right attention sinks? We investigate both the initialisation strategy and the number attention sinks, with results shown Figure & Using only 64 VOC images sufficient predict how an initialisation strategy will perform on the full LVIS MiniVal set, enabling rapid evaluation of many candidates. This is especially important since no single "one-sizee-ftt--l" strat- egy works across models. LLMDet, in particular, diverges from other early-fusion models, likely due to its LLM-aided training altering vision laniuage interactions.
Figure g (right) shows the effect of the number of sinks for LLMDet-T. Here, 24 attention sinks perform best on LVIS MiniVal, while gains diminish sharply after 8. Across all models, using more than the optimal number never harms detection performance. Multiple sinks can even share the same initial embedding, as positional encodings introduce sufficient variability.
7 RELATED WORK
Open-Vone Vocabulary Detectors (OVD) were first introduced an extension of CLIP (Radford et al.,
1 to object detection, as explored by (Gu et al. 2022). Building on this idea, GLIP (Li et al, took different approach by training
object detector from scratch rather than adapting a pre-trained CLIP model. Since then, both directions have seen significant advancements, with models such OWL (Minderer al. (022). OWLv2 (Minderer al. 2023), and the Grounding DINO family (Liu et al.] 2024b] Ren et al., 2024b a). The current state of the art among openly available models is set by OV-DINO (Wang et al. 2024b) and I.LMDet (Fu et al. 2025). More
2022
8

---

Published as a conference paper at ICLR 2026
LVS Kinival VOCao
(dif (difeernn attention sink Inst alliaiions
Inser
0.4
5.275
0.3
(45 200-New 68.
6.250
6.255
0.39
0.1
5.150
0.25
6.30
0.35
0.40
0.05
0.50
0.55
0.0
V0Caa APaur
225
24 SPAS
12 shiks
+ 48 sirks
15ck
sinss
Sirkk
Figure 8: Left: Comparing different attention sink initialisation strategies and the corresponding performance on 64 images (X-axis) of the 20 class VOC dataset vs. the full 1203 class LVIS Mini dataset (y-axis). There strong positive linear correlation, meaning a small number of images of relatively narrow dataset can accurately predict the performance of attention sinks on much broader range of classes. Right: Exploring the effect of the number of attention sinks on FPR using LLMDet-T. Performance gains start diminishing from about 8 sinks, with the optimal number of attention sinks being around 24 tokens.
recently, OVDs have also been streamlined for real-time applications, exemplified by YOLO-World (Cheng et al. 2024), YOLOE (Wang et al. 2025), and OmDet Turbo (Zhao al. 2024a).
Vision-Language Hallucination Because false positives in OVD models arise due vision- language fusion operations, there are some similarities with research on hallucination prevention for large vision-language models. Earlier work focused on methods such instruction tuning (Wang
et al. 2024a Liu et al. 2024a) post-hoc remedies such as hallucination detection and correction (Gunjal et 2024|] Yin et al. 2024), but more recent work has identified issues in attention opera-
tions which focus mainly on preceding text instead of vision tokens (Sarkar et al. 2025 Yang al. 2025
Liu et al. 2025 Min et al. 2025).
A similar pattern arises for open-vocabulary detection, where newer Grounding DINO models at- tempt to mitigate false positives through training on additional negatives (Ren et al. 2024b), prac- titioners have started applying post-hoc methods such as verification of OVD detections with CLIP models (GitHub Issue 2023). and in this paper we identify and propose a remedy for issues in vision-language attention operations.
Concurrent work in computer vision (Darcet et al. 2023) and natural language processing (Xiao et al. 2023) has shown the propensity of large transformer models repurpose input tokens as sort of scratch pad for calculations that are not necessarily related to the specific tokens in question. In the case of vision transformers, this may lead issues when certain assump- tions are made, such as feature locality for dense prediction, which does not hold for the scratch pad tokens (Darcet al. 2023). The introduction of attention sink tokens, learnt or constructed (Tiang al. 2025), can redirect scratch pad attention to special tokens unrelated to the input image. This keeps feature locality, while retaining the benefits of scratch pad tokens. For Xiao et al. (2023), the problem the deterioration of long-context language modelling once certain attention sink tokens exit the context window, with the solution to simply always keep those tokens the context.
Attention Sinks
Instead of re-routing or keeping track of existing information, utilise attention sinks seman- tically neutral attention source. This allows the model to pick sink tokens over target classes none of the target classes is present in the image.

---

Published as a conference paper at ICLR 2026
8 CONCLUSION
We have identified and quantified the pervasive issue of false positive predictions on background images for open-vocabular, detectors with early-fusion architectures. We identified their vision- language fusion layers as the cause of these errors, specifically the fact that these layers do not have the ability ignore all target class tokens in the prompt when none are relevant. We proposed training- fereing-free attention sinks as cheap and easy way to mitigate the issue, attention sinks improve AP up 5x at low false positive rates. Although we were unable find mime-size-iit--at solution in terms of attention sink initialisation strategy, we showed that candidate strategies can be efficiently evaluated on a small set of images. We encourage future work on training open-vocabulary detectors with carly-fusion architectures from scratch to incorporate gated attention similar option that explicitly allows the model to discard irrelevant class information.
REPRODUCIBILITY STATEMENT
All details needed to reproduce our results, including a stand-alone code snippet, can be found in Section 5 and Appendix A.I
REFERENCES
Chris Alberti, Jeffrey Ling, Michael Collins, and David Reitter. Fusion of detected objects in text for visual question answering, 2019. URL
05054
Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, Zheng Zhang, Dazhi Cheng, Chenchen Zhu, Tianheng Cheng, Qijic Zhao, Buyu Li, Xin Lu, Rui Zhu, Yue Wu, Jifeng Dai, Jingdong Wang, Jianping Shi, Wanli Ouyang, Chen Change Loy, and Dahua Lin. MMDetection: Open mmlab detection toolbox and benchmark. arXiv preprint arXiv: / 1906:00900..55 2019.
Tianheng Cheng, Lin Song, Yixiao Ge, Wenyu Liu, Xinggang Wang, and Ying Shan. Yolo-world: Real-time open-vocabulary object detection. In Proc. IEEE Conf. Computer Vision and Pattern Recognition (CVPR), 2024.
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers, 2023.
Mark Everingham, Lue Van Gool, Christopher K. Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge, 2010.
Shenghao Fu, Qize Yang, Qijie Mo, Junkai Yan, Xihan Wei, Jingke Meng, Xiaohua Xie, and Wei- Shi Zheng. LImdet: Learning strong voca-bucy object detectors under the supervision of large language models. arXiv preprint arXiv:2501.11854. 2025.
GitHub Issue. Many false positives, 2023. URL https:://gttuub.comm www.e//m/mIEAAReeeeeeee
Yash Goyal, Tejas Khot, Douglas Summers-Stay Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Xiuye Gu. Tsung- Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulary object detection via vision and language knowledge distillation. In International Conference on Learning Representations, 2022.
Anisha Gunjal, Jihan Yin, and Erhan Bas. Detecting and preventing hallucinations in large vision language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 18135-18143, 2024.
Agrim Gupta, Piotr Dollar, and Ross Girshick. LVIS: A dataset for large vocabulary instance seg- mentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019.
10

---

Published as a conference paper at ICLR 2026
Shuting He, Henghui Ding, Chang Liu, and Xudong Jiang. GREC: Generalized referring expression comprehension. arXiv preprint ArXiv:2308. 16182, 2023.
Nick Jiang, Amil Dravid, Alexei A Efros, and Yossi Gandelsman. Vision transformers don't need trained registers. In arXiv preprint arXiv:2506.0800 08010, 2025.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferltGame: Referring to objects in photographs of natural scenes. In Alessandro Moschitti, Bo Pang, and Walter Daele- mans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 787-798, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1D14-11000 1086. URL /Lliittts:/aaaaaancllloooo org
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, Kai-Wei Chang, and Jianfeng Gao.
Grounded language-image pre-training. In CVPR, 2022.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. The 2023 Conference on Empirical Methods in Natural Language Processing, 2023. URL /ttp::s:///perreeeew net. forum? ?id=
xoz2w0XZXF
Tsung- Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitncck, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Mitigating hallucination in large multi-modal models via robust instruction tuning. In ICLR, 2024a.
Sheng Liu, Haotian Ye, and James Zou. Reducing hallucinations in large vision-language models via latent space steering. In The Thirteenth International Conference on Learning Representations, 2025. https:///ppeeeeeee...tttttt/te
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pp. 38-55. Springer, 2024b.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Kyungmin Min, Minbeom Kim, Kang-il Lee, Dongryeol Lee, and Kyomin Jung. Mitigating hal- lucinations in large vision- vision-language language models via summary-guider decoding. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Findings of the Association for Computational Linguistics: NAACI 2025, pp. 4183-4198, Albuquerque, New Mexico, April 2025. Association for Compu- tational Linguistics. ISBN 891-98889977--9955 doi: 18.18653/v 1/2025. findings-naacl.235. URL
https::/eclanttioogg..orgg (tt.::22....iiiigg findings naacl. 235/
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovitskiy, Aravindh Mahendran, Anurag Arna,, Mostafa Dehghani, Zhuoran Shen, Xiao Wang, Xiaohua Zhai, Thomas Kipf, and Neil Houlsby. Simple open-vocabulary object detec- tion with vision transformers. ECCV, 2022.
Matthias Minderer, Alexey Gritsenko, and Neil Houlsby. Scaling open-vocabulary object detection. NeurIPS, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICMI., 2021.
Tianhe Ren, Yihao Chen, Qing Jiang, Zhaoyang Zeng, Yuda Xiong, Wenlong Liu, Zhengyu Ma, Junyi Shen, Yuan Gao, Xiaoke Jiang, Xingyu Chen, Zhuheng Song, Yuhong Zhang, Hongjie Huang. Han Gao, Shilong Liu, Hao Zhang, Feng Li. Kent Yu, and Lei Zhang. Dino-x: A uni- fied vision model for open-world world object detection and understanding, 2024a. URL https:
7/arxiv v. wttpw/ww.../////211111.. 14347
11

---

Published as a conference paper at ICLR 2026
Tianhe Ren, Qing Jiang, Shilong Liu, Zhaoyang Zeng, Wenlong Liu, Han Gao, Hongjie Huang, Zhengyu Ma, Xiaoke Jiang, Yihao Chen, Yuda Xiong, Hao Zhang. Feng Li, Peijun Tang, Kent Yu, and Lei Zhang. Grounding dino 1.5: Advance the "edge" of open-set object detection, 2024b.
Frank Ruis, Gertjan Burghouts, and Hugo Kuijf. Textual inversion for efficient adaptation of open- vocabulary object detectors without forgetting, 2025. URL htttps://axxv..ortt abs/
2508. 05323
Pritam Sarkar, Sayna Ebrahimi, Ali Etemad, Ahmad Beirami, Sercan O Arik, and Tomas Pfis- ter. Mitigating object hallucination in MLLMs via data-augmented phrase-level alignment. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
7/ aeeenrevieww.ntt/ormnnet
Ew8iq2:
Andrew Shin, Masato Ishii, and Takuya Narihira. Perspectives and prospects on transformer ar- chitecture for cross-modal tasks with language and vision. Int. Comput. Vision, 130(2): 435-454, February 2022. ISSN 0920-5691. doi: 10.1007/s11263 1007//111663022--1547777
http::::
1007/011262 021 01547-8
Ao Wang, Lihao Liu, Hui Chen, Zijia Lin, Jungong Han, and Guiguang Ding. Yoloe: Real-time seeing anything. arXiv preprint arXiv:2503.07450 2025.
Bin Wang, Fan Wu, Xiao Han, Jiahui Peng, Huaping Zhong, Pan Zhang, Xiaoyi Dong, Weijia Li, Wei Li, Jiaqi Wang, et al. Vige: Visual instruction generation and correction. In Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 5309-5317, 2024a.
Hao Wang, Pengzhen Ren, Zequn Jie, Xiao Dong, Chengjian Feng, Yinlong Qian, Lin Ma, Dongmei Jiang, Yaowei Wang, Xiangyuan Lan, et al. Ov-dino: Unified open-vocabuse vocabulary detection with language-aware selective fusion. arXiv preprint arXiv:2407.07844 2024b.
Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. Efficient streaming language models with attention sinks. arXiv, 2023.
Tianyun Yang, Ziniu Li, Juan Cao, and Chang Xu. Mitigating hallucination in large vision- language models via modular attribution and intervention. In The Thirteenth International Confer- ence on Learning Representations, 2025. URL Bjq4W7P2Us
Shukang Yin, Chaoyou Fu, Sirui Zhao, Tong Xu, Hao Wang. Dianbo Sui, Yunhang Shen, Ke Li, Xing Sun, and Enhong Chen. Woodpecker: Hallucination correction for multimodal large lan- guage models. Science China Information Sciences, 67(11):220105, 2024.
Tiancheng Zhao, Peng Liu, Xuan He, Lu Zhang, and Kyusong Lee. Real-time transformer-based open-vocabulary detection with efficient fusion head. arXiv preprint 2rXiv:2403.06892 2024a.
Xiangyu Zhao, Yicheng Chen, Shilin Xu, Xiangtai Li, Xinjiang Wang, Yining Li, and Haian Huang. An open and comprehensive pipeline for unified object grounding and detection. arXiv preprint arXiv:2401.02361. 2024b.
Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for languae models. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
A APPENDIX
A.1 IMPLEMENTATION
The implementation is very straightforward. All that is needed to implement training freee attention sinks for an MM-Grounding DINO model in mmdetection, for example, is the following:
12

---

Published as a conference paper at ICLR 2026
from mmdet... apis import DetInferencer
2
3 inferencer DetInferencer (mode1= py', weights='Ben paaas.pphh
5 nsi 24
si [f" [unused(i]]" for in range (nsi)]
inferencer. model.
wttp://eeeeeoteeeeeeee.aaaaooooo (si)
word cord_embeddings. woight, data 10 [[l:nsi+1] E.mean (dim=0) unsqueeze (0) repeat (nsi, 1)
E : inforencor.mode wtw.mmdd....nggggemmee....e
classes ["tractor"]
result inferencer ("dog. j,g", show-True, texts=classes si)
filter out predictions using sinks
16 keep = [e < (elassess) for e in result l'predictions! [0] l'labels!] 111 17 result [k: v for i, (k, v) in enumerate (result. srreult.itemss ()) if keep [i]
Here E is shorthand for the word embedding weights of the Embeddings module of the language backbone. By appending the si array to your prompt, attention sinks will be enabled, and false positives will be mitigated. All that is left do after inference is to discard any predictions with one of the attention sinks as the predicted class.
A.2 CHOICE OF INITIALISATION STRATEGY
Intuitively, a good training-free free attention sink should have a high enough activation with any vision token to overrule irrelevant classes in the prompt, but not too high to impact positive class. Some options we evaluated are as follows:
During training, the vision backbone should have learnt to increase the embed- ding distance to irrelevant classes and lower the distance to relevant classes. This may in turn result completely random embeddings ending up somewhere in-between. We find that this initialisation strategy performs best for the original Grounding DINO models.
Random:
Mean:
The mean of all word embeddings in the language model vocabulary was chosen for similar reason as the random initialisation, with the extra reasons being that it is very explicitly average and there is no chance of "bad luck" resulting in suboptimal random initialisation. find that this initialisation strategy performs best for both GLIP and MM- EMM-Grouding DINO models.
Special Characters such as parentheses, exclamation points, etc.: Neither of the previ- ous initialisation strategies work well for LLMDet, which we hypothesise be caused by language-languse interactions throwing the model off when the attention sinks are too far removed from what has been seen during the LLM-aided training. Special characters, specifically brackets and parentheses, any object in particular. We find that for LLMDet, "[O]" performs best as an attention sink initialisation (which is one sink consisting of 4 tokens).
often be found in sentence, but do not relate to
A.3 ADDITIONAL EXPERIMENTS
Are background false positives more common for common classes? To explore whether com- mon classes drive the effect of background false positives, we split the LVIS dataset into 57 COCO- overlapping classes (our proxy for "common") and the remaining 1,146 classes (our "long tail"). Surprisingly, the results are nearly identical: both groups average 2.3 false positives per class per image, with standard deviations of 0.55 (LVIS) and 0.81 (COCO). This, combined with the observa- tion that even entire negative phrases can trigger false positives (Figure I suggests that the specific content of the negative prompt does matter.
13

---

Published as a conference paper at ICLR 2026
Instead, find different pattern. False positives per negative prompt show moderate positive correlation (Pearson 7 = 0.34) with the number of medium-to-large objects in the image (annotation area 2 64 Because LVIS annotations
-xheuuxaus,,ve, there are usually more large objects present than lahelled, so the observed correlation may be an underestimate the true relationship. This points likely explanation: the model learns propose class-agnostic bounding boxes, which are then randomly assigned negative classes, since the corresponding features contain negative class information (Figure
Results on POPE Table 2 shows the results of carly-fusion models, without and with attention sinks, on the POPE benchmark. All models again see large AP gains at low FPRag except for Grounding DINO. This is because Grounding DINO has been trained on COCO, which has allowed it to properly calibrate its predictions on this specific combination of images and classes. However, as seen in Table [] Grounding DINO shows significant degradation on LVIS at low FPR even though the images of both benchmarks are identical. This further shows the impracticality of han- dling background false positives through training on explicit negatives alone, it does not generalise to other classes without fixing the vision- vision-language fusion layers.
Model
APPRR
0.0.
AP
AP
AP
Early-Fusion Fusion
GLIP-T
0.092
0.302
0.369
0.436
0.507
GDINO-T
0.293
0,413
0.457
0.505
0,546
MM-GDINO-T
0.107
0.262
0.337
0.432
0.552
LLMDet-T
0.177
0.278
0.349
0.432
0.59
Early-Fusion altn sinks
GLIP-T
0.178 (+0.086)
0,35 (+0.048)
0.407 (+0.038)
0.458 (+0.022)
0,506 (-0.001)
GDINO-T
0.308 (+0.015) 0.413 (+0.0)
0.463 (+0.006)
0.5 (-0.005)
0.54 (-0.006)
MM-GDINO-T
0.179 (+0.072) 0,347 (+0.085) 0.419 (+0.082) 0.476 (+0.044)
0,523 (-0.029)
LLMDet-T
0.202 (+0.025) 0.417 (+0.139) 0.458 (+0.109) 0.488 (+0.056) 0.519 (-0.071)
Table 2: Results on the POPE hallucination detection benchmark, adapted to object detection, five levels of FPRng. Results for Grounding DINO are indicated in gray because the model has been trained on COCO, which is the source of POPE's images and annotations. This does further show that training can mitigate background false positives for specific datasets, but does not scale to other datasets (indicated by significantly worse results of GDINO on LVIS, even though the images are the same as POPE).
A.4 OTHER ATTEMPTS
In this section, we briefly touch on several experiments that did not perform as well as the methods in the main text, but might nonetheless be interesting for future work this topic.
Training attention sinks For all models evaluated with attention sinks in the main text, have additionally tried training the attention sinks through textual inversion (Ruis et al.) 2025). form of prompt tuning where we only update the attention sink tokens and keep the rest of the model weights fixed. We used the VOC dataset (Everingham et al. (010)) training data and generated background-only negatives as described in Section []] We additionally experimented with loss in which we explicitly mark the attention sinks as the background class. We find that this does not perform better, and some cases even worse, compared to a training-free random or mean initialisation of the attention sinks.
Disabling problematic attention heads is possible to identify the attention heads that contribute the most to false positive predictions, e.g. by determining which attention heads have the largest activations negative classes. We experimented with disabling smoothing activations for such attention heads. Although this did result in performance gains, is much less effective than attention sinks, and has relatively large impact positive predictions well.
14

---

Published as a conference paper at ICLR 2026
Test-time augmentations An interesting way mitigate false positives is to utilise the fact that the presence of a true positive will help in the calibration of model predictions. One can copy a visual objects present in the prompt the margins of the input image,
example of an image of one e.g. an image of a tractor when attempting to detect tractors (cf. Figure This effectively turns the backgroundoolyy image into image containing least true positive example, and predictions on the example image can be thrown away before evaluating. While we find that this is an effective approach for prominent objects, fails mitigate false positives smaller more difficult see objects. It also comes with the limitations that one needs to know a-priori which objects can be prompted for and have relevant example image ready, diminishing the open-vocabulary uses of the model, and increases the input resolution of the image.
An example of test time augmentation (TTA) that effectively mitigate the false positive prediction "tractor" without affecting the true positive "dog". Placing the example in the margin outside of the image would alleviate the need find empty spot, and allows for cropping the image back to the original resolution while throwing away predictions outside of the image to filter TTA predictions out of the results.
Figure 9:
SraCEOF:
Grounding DINO Brocton
Grounding DINO 1.5 Pro genga bloks.
Grounding DINO hint
Figure 10: Examples confident false positives for Grounding DINO models, including the api- only models 1.5 Pro and X. Despite significantly increasing the proportion of negative samples during pre-training, 1.5 Pro and X still suffer from background-only false positives.
15


Reviews:
[{"review_id": "VC0YgRZQqW", "reviewer": "Reviewer_9wAb", "summary": "This paper shows a flaw in real-world scenarios for OVD, frequently make high-confidence false positive predictions on images that do not contain the target object. For instance, prompting an OVD for \"tractor\" on an image of a dog often results in a confident \"tractor\" detection. This issue is masked by standard benchmarks like COCO and LVIS, which rarely contain images completely lacking the target classes.The authors identify that this issue specifically affects early-fusion architectures, where vision and language features are combined early via cross-attention. Late-interaction models (e.g., CLIP) do not suffer this significant deviation in performance on background images. To mitigate this without costly retraining, the authors propose a simple, training-free method, appending \"attention sink\" tokens to the input prompt to act as \"none of the above\" type tokens. The authors evaluated this method on six early-fusion models using modified LVIS and POPE benchmarks.", "questions": "See weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper identifies a major issue in real-world deployments of Open-Vocabulary Detectors (OVDs), the false positives were previous unknown.. It also explains why this issue has been missed, noting that standard benchmarks like COCO and LVIS don't show the problem because the images rarely don't contain the target.\n\nThe analysis is good. They demonstrate that these layers cannot \"pick zero tokens\" when a match is absent, forcing them to attend to irrelevant tokens. They also clearly differentiate this behavior from late-interaction models, which do not suffer from this significant deviation on background images.\n\nThe proposed solution, appending attention sink tokens to the prompt is nice, and training-free and works out-of-the-box for all evaluated models. The implementation is highlighted as very straightforward, requiring only a few lines of code.\n\nThe attention sink approach reduces background false positives, and the results are strong.\n\nThe paper proposes simple adaptations to existing benchmarks to quantify this issue, such as leveraging negative labels in LVIS to calculate a specific background false positive rate.", "weaknesses": "1. Limited Exploration of \"Attention Sink\" Design Space and Generalizability. While the proposed \"attention sink\" method is elegant and effective, the paper's exploration of its design space is limited, there could be more options explored and ablations done. However, it does work, so this is not a major weakness.\n\n2. The authors use random initializations, mean embeddings, and special characters. Do these different types of sinks perform equally across different model architectures (e.g., are Grounding DINO's cross-attention layers equally receptive to a random vector vs. a mean embedding)?\n\n3. The paper does not thoroughly explore the impact of the number of sinks. Is one sink token sufficient, or is there a diminishing return after a certain number? This is critical for practical deployment, as more tokens increase inference time.\n\n4. How does the token length of the class prompt (e.g., \"a single dog\" vs. \"a fantastic black-and-white tractor-dog running across a field\") interact with the optimal number or type of sink tokens?\n\n5. The core goal is to reduce False Positives (e.g., detecting a tractor on a dog), which is a True Negative case (no tractor is present). While the results show gains in AP/FPR on background images, the paper lacks a detailed analysis of potential collateral damage to True Positives (detecting the dog when the dog is actually present).\n\n6. The mechanism works by providing an alternative to irrelevant class tokens. A key concern is whether a confident true detection, where the vision features were already weak, is accidentally routed to the sink instead of the correct class. A sensitivity analysis showing how the confidence of marginal correct detections changes would strengthen the conclusion.\n\n7. The paper claims a training-free solution, but the conceptual problem of \"null-class\" or \"background\" modeling is not new in object detection. For instance, how does the sink mechanism relate to the concept of a \"background class\" token used in traditional detectors (like a final output class)? While the OVD context is different, clarifying the relationship to this established concept would enhance the paper's grounding in detection literature.\n\n8. The paper states that late-interaction models (like CLIP-based methods) do not exhibit the same catastrophic failure mode. However, this conclusion is based on a limited scope.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nfnXNWg2vP", "reviewer": "Reviewer_dGwH", "summary": "The paper studies zero-shot open-vocabulary detection results of foundational detectors when fed with counterfactual prompts, i.e., the prompt contain objects that do not exist in the test image. The paper finds that early-fusion detectors (i.e., fusing features of visual and textual input) output incorrect detections in front of counterfactual prompts, while late-fusion detectors are quite robust. The paper argues that early-fusion detectors are still useful in other tasks such as VQA and hence focus on early-fusion detectors including GroundingDINO and LLMDet. The paper refers to the literature and proposes to add non-learned sink tokens to improve the early-fusion detectors. The paper uses popular datasets such as COCO and LVIS to validate the proposed approach.", "questions": "The reviewer asks the authors to address each point in weaknesses listed above and does not repeat them in this Questions box.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "Below are notable strengths of this paper.\n- The paper studies foundational detectors' performance with counterfactual prompts. This is an interesting topic.\n- The solution by adding non-learned sink tokens is simple.\n- The choice of using negative annotation in LVIS images in experiments is interesting.", "weaknesses": "Below are notable weaknesses of this paper.\n\n- The motivational example in Figure 1 is not convincing. The reviewer took this image from Figure 2 and tested GroundingDINO: when using the \"tractor\" prompt, GroundingDINO actually did not output anything, which should be deemed a good thing. Yet, Figure 1 shows a rather high-confidence detection (78.5) on \"tractor\", contradicting what the reviewer observed. The paper is expected to provide more examples. Further, the paper should provide more details; for example, the rsolution and aspect ratio of the three images are different in Figure 1. Do these factors affect the detection results?\n\n- Although the paper does not mention the word \"counterfactual\", there are previous works on counterfactual textual grounding, counterfactual VQA, counterfactual visual reasoning for segmentation, etc. These works share a common point as the reviewed paper that they all analyze models' predictions for counterfactual input. However, the paper does not note these related works.\n\n- The solution by adding sink tokens are not clear. One reason is that the paper does not provide important details to understand how it works and why it can work. A diagram can help a lot. Another reason is that the paper seems to apply the sink token technique which has published by previous works. This makes the technical contribution limited. \n\n- The visual demonstrations are quite limited. Figure 1 only provides one motivational example with three images, which are expected to be the same but actually differ in resolution and aspect ratio. Figure 5 does not show texts clearly. Figure 3 is confusing -- isn't it a good thing to obtain high LVIS MiniVal AP with decreased AP FPR (with fixed threshold 0.05)?\n\n- The code snippet is hard to read. It is not pseudo code. It contains too much coding details and notations without comments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RWy5dkpEpt", "reviewer": "Reviewer_h5nZ", "summary": "This paper investigates a critical issue in OVD—the tendency of early-fusion multimodal models to produce high-confidence false positives on background images (termed background hallucinations). The authors show that conventional benchmarks like COCO and LVIS rarely include pure background images, thereby masking this pathology. Through a comparative study across early-fusion (e.g., GLIP, Grounding DINO, LLMDet) and late-interaction (e.g., OWL-ViT, DetCLIP) architectures, the paper finds that early-fusion models exhibit strikingly higher false-positive rates when presented with images containing no objects of interest. To mitigate this, the authors propose a training-free “attention sink” mechanism: augmenting the text prompt with semantically neutral sink tokens that attract residual attention and absorb irrelevant activations. Extensive experiments across six state-of-the-art OVD models demonstrate that this simple modification drastically reduces background false positives (e.g., up to 5× AP improvement at FPR=0.01 on LVIS). The proposed method requires no retraining, minimal code changes, and is compatible with existing model checkpoints.", "questions": "Have the authors considered hybrid approaches combining sink tokens with confidence calibration or background prompts?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- Novel perspective: The paper introduces “background-only” testing to reveal systematic bias in OVD robustness.\n\n- Effective and simple method: The attention sink approach is easy to implement and achieves strong performance improvements.\n\n- Strong empirical validation: Results across six early-fusion models and multiple datasets show good generality.\n\n- Reproducible and transparent: The authors provide code, metrics, and data splits for easy verification.", "weaknesses": "- Lack of theoretical grounding: The paper provides no formal model or proof explaining how attention mass redistribution through sink tokens reduces false positives.\n\n- Missing comparison with alternative methods: Only architectural baselines are considered; other hallucination mitigation strategies are not evaluated.\n\n- Limited dataset diversity: Experiments rely on curated datasets, leaving uncertainty about performance under domain shifts or in real-world, unlabeled settings.\n\n- Model-specific tuning required: The optimal number, initialization, and placement of sink tokens vary across models, making the approach not fully plug-and-play.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p35fn4UxPL", "reviewer": "Reviewer_tkvq", "summary": "This paper highlights a critical limitation in Open-Vocabulary Object Detectors (OVDs): they often generate confident false positives on images that contain no relevant objects (for example, detecting a tractor in a photo of a dog).\nThis issue is overlooked by standard benchmarks such as COCO and LVIS, which rarely include background-only images.\nThe authors show that early-fusion models (e.g., GLIP, Grounding DINO) are particularly vulnerable to this problem due to information leakage in cross-modal attention, whereas late-interaction models (e.g., CLIP, OWL-ViT) are more robust.\nTo address this, the paper introduces a simple, training-free method that appends attention sink tokens to prompts, effectively reducing background activations and false detections without additional training.\nExperimental results demonstrate consistent improvements across multiple OVD architectures.", "questions": "- Do attention sinks affect true positive recall when target classes are present?\n- What is the computational overhead in real deployment?\n- Can the method generalize to domain-shifted or web-scale datasets?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Clear identification of the background false positive problem ($\\text{FPR}_\\text{bg}$) in early-fusion OVD models.\n\n- Strong quantitative analysis using LVIS negative labels and adapted POPE benchmark.\n\n- Simple, training-free solution (attention sink tokens) effectively reduces $\\text{FPR}_\\text{bg}$ and restores $\\text{AP}$.\n\n- Consistent improvements across multiple models with minimal implementation cost.", "weaknesses": "- Ablation limited to initialization and count; lacks analysis of token placement or interaction.\n- No runtime or efficiency analysis for large-scale inference.\n- Explanation mostly empirical without theoretical grounding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)